Video analysis support device and program

Abstract

To support managers in easily checking whether operations are being performed correctly from surveillance camera footage.SOLUTION: An acquisition unit acquires video data captured by an imaging device that captures a space in which a business is performed. A setting unit accepts settings of business rules indicating detection conditions that define a state in which a business is performed correctly or a state in which a business is not improperly performed in a frame of the video data. A determination unit determines whether a frame that satisfies the detection conditions is present in the video data. A generation unit generates index data that indicates a frame that meets the detection conditions.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a moving image analysis support device, a moving image analysis support system, and a program.

Background Art

[0002] In order to monitor whether any illegal act has been committed in a certain space, a surveillance camera for imaging the space may be provided. The video captured by the surveillance camera is recorded. Thereby, when there is a suspicion of an illegal act, the administrator can confirm the presence or absence of the illegal act by playing back the recorded moving image data.

[0003] Patent Document 1 discloses a technique for determining a state from an image captured by a surveillance camera, monitoring whether any illegal act has been committed, and making an announcement according to the state, and a technique for confirming an illegal act by playing back the recorded moving image data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, the moving image data captured by the surveillance camera often has a long recording time, and visually checking the moving image data to identify an event such as an illegal act places a great burden on the administrator. An object of the present invention is to provide a moving image analysis support device, a moving image analysis support system, and a program that can assist an administrator so that it is possible to easily confirm whether a business is being properly conducted from the video of a surveillance camera.

Means for Solving the Problems

[0006] According to a first aspect of the present invention, the motion image analysis support device includes: an acquisition unit that acquires motion image data captured by an imaging device that images a space in which work is performed; a setting unit that accepts the setting of work rules indicating detection conditions that define a state in which the work is performed correctly or a state in which the work is not performed correctly in the frame of the motion image data; a determination unit that determines whether or not there is a frame in the motion image data that satisfies the detection conditions; and a generation unit that generates index data indicating the frame specified in the detection conditions.

[0007] According to a second aspect of the present invention, the video analysis support system comprises a video analysis support device according to the above aspect and a playback unit that plays back the video data based on the index data.

[0008] According to a third aspect of the present invention, the program causes a computer to perform the following steps: acquire video data captured by an imaging device that captures a space in which work is performed; accept the setting of a business rule that indicates detection conditions that define whether the work is performed correctly or not in a frame of the video data; determine whether there is a frame in the video data that satisfies the detection conditions; and generate index data that indicates the frame specified in the detection conditions. [Effects of the Invention]

[0009] According to at least one of the above embodiments, it is possible to assist administrators in easily verifying from surveillance camera footage whether work is being performed correctly. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the configuration of the video analysis support system according to the first embodiment. [Figure 2] This is a schematic block diagram showing the configuration of the video server according to the first embodiment. [Figure 3]This diagram shows the structure of the analysis results file according to the first embodiment. [Figure 4] This is a schematic block diagram showing the configuration of a client terminal according to the first embodiment. [Figure 5] This is a flowchart showing how to set business rules using the video analysis support system according to the first embodiment. [Figure 6] This figure shows an example of the analysis target selection screen according to the first embodiment. [Figure 7] This figure shows an example of a business rule setting screen according to the first embodiment. [Figure 8] This flowchart shows the analysis processing method by the video server according to the first embodiment. [Figure 9] This is a flowchart showing a method for playing back video data using a video data analysis support system according to the first embodiment. [Figure 10] This figure shows an example of a video selection screen according to the first embodiment. [Figure 11] This figure shows an example of a playback screen according to the first embodiment. [Figure 12] This is a schematic block diagram showing the configuration of a computer according to at least one embodiment. [Modes for carrying out the invention]

[0011] <First Embodiment> Configuration of the video analysis support system The embodiments will be described in detail below with reference to the drawings. Figure 1 is a diagram showing the configuration of the video image analysis support system 1 according to the first embodiment. The video image analysis support system 1 assists administrators in monitoring for the presence or absence of fraudulent activity. Specifically, the video image analysis support system 1 assigns an index to points in the video image data captured by the surveillance camera 11 that may indicate fraudulent activity, and plays back the video image data based on the index. This allows administrators to focus their monitoring on points in the video image data that may indicate fraudulent activity.

[0012] The moving image analysis support system 1 includes a monitoring camera 11, a moving image server 13, and a client terminal 15. The monitoring camera 11 is provided to image the space where business is conducted, that is, the space to be monitored. For example, the monitoring camera 11 is provided in a bank branch. The moving image server 13 stores the moving images captured by the monitoring camera 11. The moving image server 13 analyzes the moving image data and generates index data indicating points where there is a possibility of illegal acts in the moving image data. The client terminal 15 reproduces the moving image data based on the index data generated by the moving image server 13. The moving image server 13 and the client terminal 15 are provided at the head office of the bank. The moving image server 13 is an example of a moving image analysis support device.

[0013] FIG. 2 is a schematic block diagram showing the configuration of the moving image server 13 according to the first embodiment. The moving image server 13 includes an acquisition unit 131, a moving image storage unit 132, a detection unit 133, a setting unit 134, a setting storage unit 135, a selection unit 136, a determination unit 137, a generation unit 138, and an output unit 139.

[0014] The acquisition unit 131 acquires moving image data from the monitoring camera 11. The acquisition unit 131 records the acquired moving image data in the moving image storage unit 132. The video storage unit 132 stores video data acquired by the acquisition unit 131 in association with an analysis result file showing the analysis results of the video data. Figure 3 is a diagram showing the configuration of the analysis result file according to the first embodiment. As shown in Figure 3, the analysis result file includes video data information, detection result data, index data, and version data. Video data information is metadata of the video data to be analyzed. Video data information includes the format of the video data, file name, file path, file size, frame rate, and number of frames. Detection result data shows information about the object detected from the video data. Detection result data includes the frame number in which the object was detected, the type of object detected, the confidence level of the detection, the coordinates of the area in which the object was detected (top-left coordinate position of the area), and the size of the area in which the object was detected (height and width). Index data shows the index of the frame in which processing according to the business rule was detected. Index data includes the frame number in which processing according to the business rule was detected, a message regarding the detected processing, a recording type indicating whether the index was assigned manually (M: Manual) or automatically (A: Automatic), and a warning level. Warning levels are indicated by three types, for example: "Level 0" indicating that no problematic processing was detected, "Level 1" indicating that processing suspected of being fraudulent was detected, and "Level 2" indicating that fraudulent activity was detected. Version data indicates the version of the analysis results file. Version data is updated whenever values ​​are added, changed, or deleted from the analysis results file.

[0015] The detection unit 133 detects an object from the moving image data acquired by the acquisition unit 131, and outputs detection result data indicating the frame in which the object is detected, the type of the detected object, the region where the object is detected, and the reliability of the detection. Specifically, the detection unit 133 inputs each frame of the moving image data into a learned object detection model, and performs object detection processing for each frame. The object detection model is a model learned to output the coordinates of the bounding box including the object included in the image and the type of the object included in each bounding box when the image is input. The object detection model is learned using a learning dataset composed of an input sample which is a frame of the moving image data and an output sample which is a combination of a bounding box surrounding the object included in the input sample and the type of the object. The type of the object to be detected is selected in advance through consideration in view of the business content as those with a high possibility of being the detection target. The detection result data indicating the detection result of the object is recorded in the moving image storage unit 132 in association with the moving image data of the detection target.

[0016] The setting unit 134 receives the setting of business rules indicating the start condition and end condition of the business, and the detection condition of the illegal act, for each space and business of the monitoring target. The start condition and the detection condition are represented by the type of the object to be detected and the detection state of the object. The detection state is selected from whether the object is detected, whether the state in which the object is detected continues, and whether the state in which the object is not detected continues. The detection state may include the designation of the region detected in the frame of the moving image data. The end condition may be represented by the detection state of the object or may be represented by the elapsed time since the start condition was satisfied. The business rules set by the setting unit 13 are recorded in the setting storage unit 135 in association with the business rule name.

[0017] The selection unit 136 receives the selection of the business rules used for monitoring each moving image data stored in the moving image storage unit 132. The determination unit 137 determines whether each frame of the video data satisfies the detection conditions of the business rules, based on the analysis results of the video data by the detection unit 133 and the business rules associated with the video data. The determination unit 137 identifies the frame numbers of the frames that satisfy the detection conditions.

[0018] The generation unit 138 generates index data that associates the frame number identified by the determination unit 137 with the detection conditions detected in the frame indicated by that frame number. The output unit 139 outputs the analysis result file stored in the video storage unit 132 to the client terminal 15.

[0019] Figure 4 is a schematic block diagram showing the configuration of the client terminal 15 according to the first embodiment. The client terminal 15 comprises a display control unit 151, a request unit 152, a playback unit 153, and an editing unit 154. The display control unit 151 controls the display based on the information received from the video server 13. The request unit 152 sends a request to the video server 13 based on the user's input. The playback unit 153 plays back the video data acquired from the video server 13. The playback unit 153 specifies the playback position of the video data by receiving an index specification from the user based on index data. The editorial unit 154 edits the index data of the analysis result files stored in the video server 13. Specifically, the editorial unit 154 adds an index indicating the playback position of the video image data being played back by the playback unit 153 to the index data.

[0020] 《How to set up business rules》 Figure 5 is a flowchart showing how to set business rules using the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 sends a request for rule information to the video server 13 (step S1). When the output unit 139 of the video server 13 receives the request for rule information from the client terminal 15, it sends the file name of the video data stored in the video storage unit 132, the analysis result file, and the business rules stored in the setting storage unit 135 to the client terminal 15 (step S2).

[0021] The display control unit 151 of the client terminal 15 generates an analysis target selection screen based on the file name of the video data received from the video server 13 and the analysis result file, and displays it on the display (step S3). Figure 6 is a diagram showing an example of the analysis target selection screen according to the first embodiment. The analysis target selection screen displays a video list i01 for selecting an analysis target from multiple video data, and a setting start button i02 for proceeding with the setting of business rules for the selected video data. The video list i01 includes a checkbox i011 for accepting the selection of whether or not to include it as an analysis target, a mark i012 indicating the analysis result, and a file name i013. The mark i012 indicates whether or not the analysis of the associated video data has been completed, and if so, whether or not fraudulent activity or activity requiring verification has been detected. If both fraudulent activity and activity requiring verification are detected, the mark i012 is displayed in a manner indicating that fraudulent activity has been detected.

[0022] The request unit 152 of the client terminal 15 determines whether or not the setting start button i02 has been pressed by the user (step S4). If the setting start button i02 is not pressed (step S4: NO), the request unit 152 returns to step S4 and waits for the setting start button i02 to be pressed. On the other hand, if the setting start button i02 is pressed (step S4: YES), the checkbox i011 associated with the video data to be analyzed is turned ON on the analysis target selection screen, and when button i02 is pressed, the display control unit 151 of the client terminal 15 generates a business rule setting screen based on the file name of the video data received from the video server 13 in step S2 and the business rule, and displays it on the display (step S5). Figure 7 is a diagram showing an example of a business rule setting screen according to the first embodiment. The business rule setting screen includes a video list i11, a rule list i12, a rule display pane i13, a rule add button i14, and an analysis start button i15.

[0023] The video list i11 displays the filenames of the video data associated with the selected checkbox i011. The video list i11 is configured to allow selection of one or more files from multiple video data files. The rule list i12 displays the names of the business rules received from the video server 13. Each business rule name is associated with a checkbox, which allows the user to choose whether or not to apply the business rule to the video data selected in the video list i11. When a business rule in the rule list i12 is clicked, the contents of that business rule are displayed in the rule display pane i13. The Add Rule button i14, when pressed, accepts the addition of a business rule. The Start Analysis button i15, when pressed, executes an analysis based on the associated business rule for each video data image to be analyzed.

[0024] The request unit 152 of the client terminal 15 determines whether or not the rule addition button i14 has been pressed by the user (step S6). If the rule addition button i14 has been pressed (step S6: YES), the display control unit 151 displays the rule creation screen on the display (step S7). The rule creation screen includes a form for inputting the rule name, start condition, end condition, detection condition, and detection accuracy.

[0025] The request unit 152 accepts the settings for the type of object to be detected from the video data and the detection state as conditions for starting the business rule. The type of object is selected from a list of types that can be detected by the object detection model used by the detection unit 133 of the video server 13. The detection state is selected from either the state in which the object has changed from not being detected to the state in which it has been detected, or the state in which the object has changed from the state in which it has been detected to the state in which it has not been detected. In addition, the setting of the frame area can be accepted as an option for the detection state. The request unit 152 accepts the setting of the elapsed time from the point when the detection state or start condition of the target object is satisfied as the termination condition of the business rule.

[0026] The request unit 152 accepts settings for the type of object to be detected from the video image data and the detection state as detection conditions for the business rule. The detection state related to the detection conditions is selected from one of the following: the object has changed from a state where it is not detected to a state where it is detected; the object has changed from a state where it is detected to a state where it is not detected; the object has not changed from a state where it is not detected to a state where it is detected until the termination condition is met; or the object has not changed from a state where it is detected to a state where it is not detected until the termination condition is met. In addition, the setting of the frame area can be accepted as an option for the detection state. The request unit 152 accepts an integer greater than 0 and less than or equal to 100 as the detection accuracy of the target object in the business rules. The detection accuracy may also be represented by a number between 0 and 1.0. Furthermore, the detection accuracy may be entered directly as a numerical value or by using a slider. In this case, the input range of the slider may be set, for example, to 70 or more and less than or equal to 100, to allow for the specification of a more precise numerical value for the detection accuracy.

[0027] When the request unit 152 receives input of a business rule from the user, it sends the input business rule to the video server 13 (step S8). When the setting unit 134 of the video server 13 receives the business rule from the client terminal 15, it records the received business rule in the setting storage unit 135, associating it with the business rule name (step S9). The video analysis support system 1 returns to step S2 and sends information including the added business rule to the client terminal 15.

[0028] If the rule addition button i14 is not pressed on the business rule setting screen (step S6: NO), the request unit 152 determines whether the analysis start button i15 has been pressed by the user (step S10). If the analysis start button i15 has not been pressed (step S10: NO), the request unit 152 returns to step S6 and monitors whether the button has been pressed.

[0029] If the analysis start button i15 is pressed (step S10: YES), the request unit 152 sends an analysis request for the video data to the video server 13 (step S11). The analysis request includes information indicating the relationship between each video data displayed in the video list i11 and the business rules indicated by the checkboxes that are ON in the rule list i12 for that video data. The selection unit 136 of the video server 13 accepts the selection of business rules to be applied to each video data upon receipt of the analysis request (step S12). The video server 13 performs an analysis of each video stored in the video storage unit 132 according to the selected business rules (step S13).

[0030] Figure 8 is a flowchart showing the analysis processing method by the video server 13 according to the first embodiment. When the video server 13 starts analysis according to the business rules, it selects video data recorded in the video storage unit 132 one by one (step S31) and executes the following processes from steps S32 to S39.

[0031] The detection unit 133 of the video server 13 determines whether or not detection result data associated with the video image data selected in step S31 exists in the video storage unit 132 (step S32). If there is no detection result (step S32: NO), the detection unit 133 uses an object detection model to detect objects from each frame of the video image data selected in step S31 (step S33). The detection unit 133 records the detection result data indicating the detection result in the video storage unit 132 in association with the video image data to be detected (step S34). For each frame, the detection result data records the frame number, the type of object detected in that frame, the coordinates of the bounding box surrounding the object, and the confidence level in association with each frame. The generation unit 138 initializes the index data associated with the video image data selected in step S31 (step S35). At this time, the generation unit 138 deletes the indexes that were added by analysis based on business rules (indexes with record type A) from the index data, but does not delete the indexes that were added manually by the administrator (indexes with record type M).

[0032] Next, the determination unit 137 selects one by one the business rules selected in step S10 as business rules to be applied to the video image data selected in step S31 (step S36), and then executes the following processes from steps S37 to S39.

[0033] The determination unit 137 identifies frames that satisfy the start conditions (start frames) and frames that satisfy the end conditions (end frames) indicated by the business rule selected in step S36, based on the analysis result file of the video data selected in step S31 (step S37). Between the identified start frames and end frames, the determination unit 137 identifies frames that satisfy the detection conditions indicated by the business rule selected in step S36, and frames that no longer satisfy the detection conditions (step S38). The determination unit 137 also identifies frames that do not satisfy the start conditions, end conditions, or detection conditions, and if the determination results differ for a predetermined number of frames (e.g., 1 frame) among consecutive frames, it considers this to be a detection omission or false detection and adjusts the determination result of the relevant frame to match the determination results of the other frames. For example, if 19 out of 20 consecutive frames satisfy the detection conditions and only 1 frame does not, the determination unit 137 treats that 1 frame as satisfying the detection conditions. The number of frames considered to be detection omissions or false detections can be changed by the administrator.

[0034] For example, if the detection condition indicates that an object has been detected, the determination unit 137 identifies the first frame indicating that an object has been detected as the frame that satisfies the detection condition. On the other hand, if the detection condition indicates that an object has been detected, and there are no frames in which an object has been detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection condition is no longer met. If the detection condition indicates that the object is no longer detected, the determination unit 137 identifies the first frame indicating that the object is not detected as the frame that satisfies the detection condition. On the other hand, if the detection condition indicates that the object is no longer detected, and there are no frames between the start frame and the end frame in which the object is not detected, the determination unit 137 identifies the end frame as the frame in which the detection state is no longer met. If the detection condition indicates that the state in which the object is detected continues, the determination unit 137 identifies the first frame, which indicates that the object is not detected, as the frame in which the detection condition is no longer met. On the other hand, if the detection condition indicates that the state in which the object is detected continues, and there are no frames in which the object is not detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection condition is met. If the detection condition indicates that the object has not been detected for an extended period, the determination unit 137 identifies the first frame in which the object was detected as the frame in which the detection condition was no longer met. On the other hand, if the detection condition indicates that the object has not been detected for an extended period, and there are no frames in which the object was detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection condition is met.

[0035] The generation unit 138 adds the index representing the frame identified in steps S37 and S38 to the index data initialized in step S35 (step S39). Each index, as shown in Figure 3, indicates the frame number, a message representing the corresponding state, the recording type, and the warning level. For example, in a frame that satisfies the detection conditions indicating fraudulent activity, the generation unit 138 sets the warning level to level 1 if the confidence level of the detected object is lower than a standard value, and sets the warning level to level 2 if the confidence level is equal to or greater than the standard value. This standard value may be, for example, a value obtained by adding a predetermined value to the detection accuracy set in the business rules.

[0036] As a result, the video server 13 can generate index data for each video image file, recording the index of the frame that corresponds to the business rule selected in step S12.

[0037] How to play video data Figure 9 is a flowchart showing the method for playing back video data using the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 sends a request for video information to the video server 13 (step S51). When the output unit 139 of the video server 13 receives the request for video information from the client terminal 15, it sends the file name of the video data stored in the video storage unit 132 and the index data associated with the video data to the client terminal 15 (step S52).

[0038] The display control unit 151 of the client terminal 15 generates a video selection screen based on the file name and index data of the video data received from the video server 13 and displays it on the display (step S53). Figure 10 is a diagram showing an example of the video selection screen according to the first embodiment. The video selection screen displays the file name i31 and a mark i32 indicating the analysis result for each video data. The video selection screen is configured to allow the selection of one of multiple video data. When one of the displayed file names i31 is selected on the video selection screen, the playback screen for the selected video data is displayed. The mark i32 indicates whether the analysis of the associated video data has been completed, and if so, whether any fraudulent activity has been detected. In other words, if there is no index data received from the video server 13, the display control unit 151 places a mark i32 indicating that the analysis of the video data has not been completed. If index data has been received from the video server 13 and there is one or more entries in the analysis result file with a warning level of 1 or 2, the display control unit 151 will mark i32 to indicate that fraudulent activity has been detected. If index data has been received from the video server 13 and all warning levels are 0, the display control unit 151 will mark i32 to indicate that no fraudulent activity has been detected.

[0039] In other embodiments, the display control unit 151 may display the mark i32 indicating the presence or absence of fraudulent activity separately for cases where there is clear fraud (when there are items with a warning level of 2) and cases where there is suspicion and requires verification (when there are no items with a warning level of 2, but there are items with a warning level of 1).

[0040] When the request unit 152 of the client terminal 15 receives a selection of one video data file from the user, it sends a request for the selected video data file to the video server 13 (step S54). The acquisition unit 131 of the video server 13 acquires the video data file from the video storage unit 132 and sends it to the client terminal 15 (step S55).

[0041] The playback unit 153 generates a playback screen for the video data and displays it on the display (step S56). Figure 9 shows an example of a playback screen according to the first embodiment. The playback screen displays a video data playback pane i51, an index pane i52, a control pane i53, and an index editing button i54.

[0042] The playback pane i51 displays video data distributed from the video server 13. If an object is detected in the displayed frame, the playback pane i51 displays a bounding box surrounding the object, as well as the object's type and confidence level. The display / hide of the bounding box can be toggled by the administrator. The index pane i52 displays a list of indices indicated by the index data associated with the video data displayed in the playback pane i51. In the index pane i52, the index is represented by a combination of playback position and message. In addition, in the index pane i52, the index is represented by a different color depending on the warning level. This allows the administrator to easily identify areas in the video data that are highly likely to contain malicious activity or areas where malicious activity is suspected. The control pane i53 displays buttons for controlling the playback of the video data displayed in the playback pane i51. Specifically, the control pane i53 displays a seek bar indicating the playback position, a back skip button to return to the playback position by a predetermined time, a stop button to stop playback, a play button to start playback, and a skip button to advance the playback position by a predetermined time. The index editing button i54 allows you to edit the index data when pressed.

[0043] The playback unit 153 receives the video data selected in step S54 from the video server 13 and plays it in the playback pane i51 (step S57). The playback unit 153 determines whether or not one of the indices displayed in the index pane i52 has been pressed (step S58). If an index has been pressed (step S58: YES), the playback unit 153 moves the playback position of the video data being played to the playback position corresponding to the pressed index (step S59). This allows the administrator to easily identify the parts of the video data that are of interest.

[0044] If the index button is not pressed (step S58: NO), the editing unit 154 determines whether or not the index editing button i54 has been pressed (step S60). If the index editing button i54 has been pressed (step S60: YES), the playback unit 153 stops playback of the video data displayed in the playback pane i51 at the current playback position. When the editing unit 154 receives a message from the administrator, it sends an index addition request to the video server 13, including the current playback position and the entered message (step S61). The generation unit 138 of the video server 13 updates the index data based on the received index addition request (step S62). This allows the administrator to supplement information such as omissions in the analysis by the video server 13.

[0045] Furthermore, if the index editing button i54 is pressed in a frame where an index already exists, the editorial department 154 accepts the modification or deletion of the index content. For example, if an administrator sees a frame with a warning level of 1, indicating suspected misconduct, and determines whether the activity detected in that frame is misconduct, they may edit the index using the following procedure: The administrator clicks on the index they want to edit, moves the playback position, and then clicks the index editing button i54. This allows the editorial department 154 to accept the edit of the clicked index. If misconduct is confirmed, the administrator changes the warning level of the index to 2; if it is confirmed that no misconduct occurred, the warning level of the index is changed to 0.

[0046] The request unit 152 determines whether the playback screen has been closed (step S63). If the playback screen has not been closed (step S63: NO), the process returns to step S55 and continues playback of the video data. On the other hand, if the playback screen has been closed (step S63: YES), the video data analysis support system 1 terminates the video data playback process.

[0047] This allows administrators to efficiently monitor operations by playing back video data using index data. Furthermore, administrators can create indexes for points of interest during video playback, which can then be used as a reference when reviewing the video data later.

[0048] 《Learning function from data requiring verification》 The administrator can determine whether a frame indicated by an index with a warning level of 1 represents fraudulent activity and rewrite the warning level based on the determination. The video server 13 may also change the business rules based on the warning levels rewritten by the administrator. For example, the video server 13 may change the start conditions, end conditions, detection conditions, and detection accuracy based on the business rules. For example, in a business rule that determines a warning level of 2 when the detection accuracy is 95% or higher, a warning level of 1 when it is 90% or higher but less than 95%, and a warning level of 0 when it is less than 90%, if the frequency of rewriting from warning level 1 to warning level 2 is high, the video server 13 may change the threshold for distinguishing between warning level 1 and warning level 2 to a value lower than 90%, for example. Also, for example, in a business rule where the end condition is 3 seconds after the start condition is met, if the detection condition relates to fraudulent activity and the frequency of rewriting from warning level 1 to warning level 2 is high, the time related to the end condition may be increased. Furthermore, for example, if the object related to the detection conditions is not related to fraudulent activity, and the warning level is frequently changed from level 1 to level 2, the time required for the termination conditions may be shortened.

[0049] Examples of setting business rules Here, we will explain an example of business rules used for analyzing video data. For example, you can set up work rules regarding cabinet operation. If the work rules for cabinet operation stipulate that cabinets should not be left open, you can set up the following work rules: The start condition for the business rule related to cabinet operation is the detection of an object of the type "open cabinet". The termination condition for the business rule related to cabinet operation is that no more objects of the type "open cabinet" are detected. The detection condition for the business rule related to cabinet operation is that no more objects of the type "person" are detected. This allows the video server 13 to monitor whether or not the worker left the cabinet open while operating it.

[0050] For example, business rules can be established regarding the exchange of documents. If the business rules for the exchange of documents stipulate receiving and storing documents submitted by customers, confirming that documents submitted to customers are stored, and detecting the loss of documents, then the following business rules can be established. The start condition for the business rules regarding the exchange of documents is the detection of two objects of the type "person" and an object of the type "document". The end condition for the business rules regarding the exchange of documents is that the two objects of the type "person" are no longer detected. The detection conditions for the business rules regarding the exchange of documents are that the state of not detecting an object of the type "document" continues, and that an object of the type "document" is no longer detected in a location other than the area where it should be stored. Note that while this business rule specifies the type of object as "document," other business rules may set conditions for other types, such as "banknotes," "coins," or "cards." Furthermore, other business rules may specify more specifically the type of document (contract, identification document, brochure, etc.) than just "document." This allows the video server 13 to monitor whether the documents were properly exchanged.

[0051] For example, operational rules can be established regarding entry and exit from the vault. If the operational rules for entry and exit from the vault stipulate that there must be two or more people entering or exiting the vault, the following operational rules can be established. The starting condition for the rules regarding entry into the vault is the detection of an object of the type "person entering." An "entering person" is, for example, a person facing inward into the vault. The ending condition for the rules regarding entry into the vault is the passage of a certain amount of time since the starting condition was met. The detection condition for the rules regarding entry into the vault is the continued absence of any object of the type "person entering" other than the object of the starting condition. The starting condition for the rules regarding leaving the vault is the detection of an object of the type "person leaving." An "person leaving" is, for example, a person facing outwards from the vault. The ending condition for the rules regarding leaving the vault is the passage of a certain amount of time since the starting condition was met. The detection condition for the rules regarding leaving the vault is the continued absence of any object of the type "person leaving" other than the object of the starting condition. This allows the video server 13 to monitor whether or not two or more people are entering or leaving the vault. Furthermore, the operational rules for entering and exiting the vault may include a detection condition that at least one of the "persons" is of the type "administrator." In other words, the detection conditions for operational rules related to entering and exiting the vault may include the condition that no object of the type "administrator" is detected.

[0052] For example, business rules can be established regarding the operation of cash dispensers. If the business rules for cash dispenser operation stipulate that tasks involving accessing the inside of the cash dispenser must be performed by both general workers and managers, the following business rules can be established. The start condition for the business rule related to cash machine operation is the detection of an object of the type "person" near the cash machine. The end condition for the business rule related to cash machine operation is that no object of the type "person" is detected near the cash machine. The detection condition for the business rule related to cash machine operation is that no object of the type "person" other than the object specified in the start condition is detected. The detection condition for the business rule related to cash machine operation may also be that no object of the type "administrator" is detected. This allows the video server 13 to monitor whether the cashier operation is being performed in the presence of an administrator. Furthermore, the above operational rules can be applied not only to cash dispensers but also to other devices such as cash management systems, important item management systems, or key management systems.

[0053] For example, as part of the operational rules, operational rules can be established regarding the placement of discarded documents into document processing devices such as shredders and document disposers. If the operational rules for placing discarded documents into document processing devices stipulate that the work should be performed by both general workers and managers, the following operational rules can be established. The start condition for the rules regarding the disposal of discarded documents is the detection of an object of the type "person" near the document processing device. The end condition for the rules regarding the disposal of discarded documents is that no objects of the type "person" are detected near the document processing device. The detection condition for the rules regarding the disposal of discarded documents is that no other objects of the type "person" are detected, in addition to the object specified in the start condition. The detection condition for the rules regarding the disposal of discarded documents may also be that no objects of the type "administrator" are detected. This allows the video server 13 to monitor whether the disposal documents are being submitted in the presence of an administrator.

[0054] Furthermore, if it is stipulated that documents be reviewed before being submitted to the document processing system, the following operational rules can be established. In this case, the start condition for the business rule regarding the disposal of documents is the detection of an object of the type "person" near the document processing device. The end condition for the business rule regarding the disposal of documents is that an object of the type "person" is no longer detected near the document processing device. The detection condition for the business rule regarding the disposal of documents is that the state in which "documents" are detected continues for a certain period of time before the object of the start condition is no longer detected in the location where an object of the type "document" should be stored before the "document" is disposed of in the document processing device. This allows the video server 13 to monitor whether or not a "person" is checking the documents before the documents are disposed of.

[0055] For example, business rules can be established regarding postmarking documents. If the task involves postmarking a specific document as proof of receipt, the following business rules can be established: The starting condition for the operational rules regarding postmarking documents is the detection of an object of the type "person" and an object of the type "document". The ending condition for the operational rules regarding postmarking documents is that no objects of the type "person" and an object of the type "document" are detected. The detection condition for the operational rules regarding postmarking documents is the detection of an object of the type "postmark" superimposed on the object of the starting condition, which is "document". This allows the video server 13 to monitor whether or not a postmark has been applied to the document.

[0056] For example, operational rules can be established regarding handover procedures at security centers, nurse stations, etc. As operational rules for handovers, the following operational rules can be established when the exchange of specific documents is performed along with roll call. The start condition for the handover work rules is the detection of two objects of the type "person" and an object of the type "document". The end condition for the handover work rules is that the two objects of the type "person" are no longer detected. The detection conditions for the handover work rules are that the state of not detecting an object of the type "person pointing" continues, and that an object of the type "document" is no longer detected in a location that is not the area where it should be stored. This allows the video server 13 to monitor whether the handover was performed properly.

[0057] Action / Effect Thus, according to the first embodiment, the video analysis support system 1 detects frames from the video data captured by the surveillance camera 11 that indicate that the work was not performed correctly, based on the work rules, and generates index data indicating those frames. This allows the administrator to review the video data based on the index data. In other words, the video analysis support system 1 can assist the administrator in easily verifying whether the work is being performed correctly from the video footage of the surveillance camera 11.

[0058] <Other Embodiments> Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes are possible. In other embodiments, the order of the above-described processes may be changed as appropriate. Also, some processes may be executed in parallel. The video server 13 according to the above embodiment may be configured by a single computer, or the configuration of the video server 13 may be divided and arranged among multiple computers, with the multiple computers cooperating with each other to function as the video server 13. In another embodiment, the video server 13 and the client terminal 15 may be configured by a single computer.

[0059] The video server 13 according to the above embodiment sets a detection accuracy for determining whether or not detection is performed for each business rule, but is not limited to this. For example, in other embodiments, the detection accuracy may be set for each type of object, or the detection accuracy may be set to a fixed value.

[0060] In the above-described embodiment, the video server 13 records the detection result data in the setting storage unit 135 when the detection unit 133 performs detection, and skips the analysis of video data for which detection result data has already been generated. This allows the time required to generate the analysis result file to be shortened by using the already generated detection result data when the detection accuracy setting in the business rules is changed after detection by the detection unit 133. On the other hand, other embodiments are not limited to this. For example, the video server 13 in other embodiments may perform detection by the detection unit 133 each time an analysis request is received. When the detection unit 133 performs detection each time in this manner, the management of the analysis result file in the video server 13 can be made easier.

[0061] The start and end conditions of the business rules according to the above embodiment are determined based on the detection results of the detection unit 133, but are not limited to this. For example, the start and end conditions of business rules according to other embodiments may be determined based on log data of devices installed in the monitored space. For example, the start condition may be that a predetermined type of cashier operation is performed on a cashier installed in the monitored space, and a frame that satisfies the start condition may be identified from the cashier's log data.

[0062] The detection conditions for the business rules according to the embodiments described above define fraudulent activity, i.e., a state in which the business was not performed correctly, but are not limited to this. For example, the detection conditions according to other embodiments may define a state in which the business was performed correctly.

[0063] <Computer Configuration> Figure 10 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, main memory 93, storage 95, and an interface 97. The aforementioned video server 13 and client terminal 15 are each implemented in the computer 90. The operation of each processing unit described above is stored in storage 95 in the form of a program. The processor 91 reads the program from storage 95, loads it into main memory 93, and executes the above processing according to the program. The processor 91 also allocates memory areas in main memory 93 corresponding to each of the storage units described above, according to the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a microprocessor.

[0064] The program may be for implementing some of the functions that the computer 90 is to perform. For example, the program may perform functions in combination with other programs already stored in storage, or in combination with other programs implemented in other devices. In other embodiments, the computer 90 may be equipped with a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also included as an example of a processor.

[0065] Examples of storage 95 include magnetic disks, magneto-optical disks, optical disks, and semiconductor memory. Storage 95 may be an internal medium directly connected to the bus of the computer 90, or it may be an external medium connected to the computer 90 via an interface 97 or a communication line. Furthermore, if this program is delivered to the computer 90 via a communication line, the computer 90 that receives the delivery may expand the program into the main memory 93 and execute the above processing. In at least one embodiment, storage 95 is a tangible storage medium that is not temporary.

[0066] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in storage 95. [Explanation of Symbols]

[0067] 1…Video analysis support system 11…Surveillance camera 13…Video server 131…Acquisition unit 132…Video storage unit 133…Detection unit 134…Setting unit 135…Setting storage unit 136…Selection unit 137…Determination unit 138…Generation unit 139…Output unit 15…Client terminal 151…Display control unit 152…Request unit 153…Playback unit 154…Editing unit 90…Computer 91…Processor 93…Main memory 95…Storage 97…Interface

Claims

1. An acquisition unit that acquires video data captured by an imaging device that images the space where work is performed, A setting unit that accepts the setting of a business rule that indicates a detection condition that defines whether the business was executed correctly or not in the frame of the video data, A determination unit that determines whether or not there is a frame that satisfies the detection conditions from the video data, A generation unit that generates index data indicating the frame defined in the above detection conditions, Equipped with, If the determination unit finds that the detection results for a predetermined number of consecutive frames are different, it will adjust the determination result of the relevant frame to match the determination results of the other frames. The aforementioned business rules include start and end conditions for the business, and the start and / or end conditions of the aforementioned business rules are determined based on log data from devices installed in the monitored space. Video analysis support device.

2. The other frame is a frame among the consecutive frames that satisfies the detection conditions. The motion image analysis support device according to claim 1.

3. The other frames mentioned above refer to the frames immediately before and after the relevant frame among the consecutive frames. The motion image analysis support device according to claim 1.

4. The motion image analysis support device according to claim 1, wherein the predetermined number is changeable.

5. On the computer, The steps include acquiring video data captured by an imaging device that images the space where work is performed, and A step of receiving the setting of a business rule that indicates detection conditions that define a state in which the business was executed correctly or a state in which the business was not executed correctly in the frame of the video data, The steps include determining whether or not there is a frame in the video data that satisfies the detection conditions, The steps include generating index data that indicates the frame specified in the detection conditions, Make it run, In the step of determining whether or not there are frames that satisfy the detection conditions from the video data, if the detection results for a predetermined number of consecutive frames are different, the determination result of the relevant frame shall be made consistent with the determination results of the other frames. The aforementioned business rules include start and end conditions for the business, and the start and / or end conditions of the aforementioned business rules are determined based on log data from devices installed in the monitored space. program.

Images