Video image analysis support device and program

Abstract

To support managers in easily checking whether operations are being performed correctly from surveillance camera footage.SOLUTION: An acquisition unit acquires video data captured by an imaging device that captures a space in which a business is performed. A setting unit accepts settings of business rules indicating detection conditions that define a state in which a business is performed correctly or a state in which a business is not improperly performed in a frame of the video data. A determination unit determines whether a frame that satisfies the detection conditions is present in the video data. A generation unit generates index data that indicates a frame that meets the detection conditions.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a video image analysis support device, a video image analysis support system, and a program. [Background technology]

[0002] In order to monitor whether or not any illegal activities are being carried out in a certain space, a surveillance camera may be installed to capture the image of the space. The images captured by the surveillance camera are recorded. In this way, when any illegal activity is suspected, the administrator can confirm whether or not any illegal activity has been carried out by playing back the recorded video data.

[0003] Patent Document 1 discloses a technology that determines the situation from images captured by a surveillance camera, monitors whether any fraudulent activity is occurring, and makes an announcement according to the situation, as well as a technology that allows fraudulent activity to be confirmed by playing back recorded video data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-124186 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, video data captured by a surveillance camera often has a long recording time, and visually checking the video data to identify events such as fraudulent activity places a heavy burden on the administrator. An object of the present invention is to provide a video analysis support device, a video analysis support system, and a program that can assist managers in easily checking whether operations are being performed correctly from surveillance camera footage. [Means for solving the problem]

[0006] According to a first aspect of the present invention, a video analysis support device includes an acquisition unit that acquires video data captured by an imaging device that captures an image of a space in which a business is performed, a setting unit that accepts settings of business rules indicating detection conditions that specify a state in which the business is performed correctly or a state in which the business is not performed correctly in a frame of the video data, a determination unit that determines whether or not a frame that satisfies the detection conditions is present in the video data, and a generation unit that generates index data indicating a frame specified in the detection conditions.

[0007] According to a second aspect of the present invention, a video analysis support system includes the video analysis support device according to the above aspect, and a playback unit that plays back the video data based on the index data.

[0008] According to a third aspect of the present invention, a program causes a computer to execute the steps of acquiring video data captured by an imaging device that captures an image of a space in which a business is performed, accepting settings of business rules indicating detection conditions that specify a state in which the business is performed correctly or a state in which the business is not performed correctly in frames of the video data, determining whether or not there is a frame from the video data that satisfies the detection conditions, and generating index data indicating frames that satisfy the detection conditions. Effect of the Invention

[0009] According to at least one of the above aspects, it is possible to assist the manager in easily checking from the video of the surveillance camera whether the work is being carried out correctly. [Brief description of the drawings]

[0010] [Figure 1] 1 is a configuration diagram of a video analysis support system according to a first embodiment. [Diagram 2] 1 is a schematic block diagram showing a configuration of a video server according to a first embodiment. [Diagram 3]FIG. 4 is a diagram showing the configuration of an analysis result file according to the first embodiment. [Figure 4] FIG. 2 is a schematic block diagram showing a configuration of a client terminal according to the first embodiment. [Diagram 5] 4 is a flowchart showing a method for setting business rules by the video analysis support system according to the first embodiment. [Figure 6] FIG. 11 is a diagram showing an example of an analysis target selection screen according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a business rule setting screen according to the first embodiment. [Figure 8] 5 is a flowchart showing an analysis processing method performed by the video server according to the first embodiment. [Figure 9] 4 is a flowchart showing a video data playback method performed by the video analysis support system according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing an example of a video selection screen according to the first embodiment. [Figure 11] FIG. 4 is a diagram showing an example of a playback screen according to the first embodiment. [Figure 12] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] First embodiment Configuration of video analysis support system Hereinafter, the embodiments will be described in detail with reference to the drawings. 1 is a configuration diagram of a video analysis support system 1 according to a first embodiment. The video analysis support system 1 supports an administrator in monitoring for the presence or absence of fraudulent activities. Specifically, the video analysis support system 1 assigns indexes to points in video data captured by a surveillance camera 11 where fraudulent activities may occur, and plays the video data based on the indexes. This allows the administrator to focus on monitoring points in the video data where fraudulent activities may occur.

[0012] The video analysis support system 1 comprises a surveillance camera 11, a video server 13, and a client terminal 15. The surveillance camera 11 is installed so as to capture images of the space where business is carried out, i.e., the space to be monitored. For example, the surveillance camera 11 is installed in a bank branch. The video server 13 stores the videos captured by the surveillance camera 11. The video server 13 analyzes the video data and generates index data that indicates points in the video data where fraudulent activity may occur. The client terminal 15 plays the video data based on the index data generated by the video server 13. The video server 13 and the client terminal 15 are installed in the bank's headquarters. The video server 13 is an example of a video analysis support device.

[0013] 2 is a schematic block diagram showing the configuration of the video server 13 according to the first embodiment. The video server 13 includes an acquisition unit 131, a video storage unit 132, a detection unit 133, a setting unit 134, a setting storage unit 135, a selection unit 136, a determination unit 137, a generation unit 138, and an output unit 139.

[0014] The acquisition unit 131 acquires moving image data from the surveillance camera 11. The acquisition unit 131 records the acquired moving image data in the moving image storage unit 132. The video storage unit 132 stores the video data acquired by the acquisition unit 131 in association with an analysis result file indicating an analysis result of the video data. FIG. 3 is a diagram showing a configuration of an analysis result file according to the first embodiment. As shown in FIG. 3, the analysis result file includes video data information, detection result data, index data, and version data. The video data information is metadata of the video data to be analyzed. The video data information includes the format, file name, file path, file size, frame rate, and number of frames of the video data. The detection result data indicates information of an object detected from the video data. The detection result data includes the frame number where the object was detected, the type of the detected object, the reliability of the detection, the coordinates of the area where the object was detected (the upper left coordinate position of the area), and the size (height and width) of the area where the object was detected. The index data indicates an index of a frame where a process according to a business rule was detected. The index data includes the frame number where a process according to a business rule was detected, a message related to the detected process, a recording type indicating whether the index was added manually (M: Manual) or automatically (A: Automatic), and a warning level. The warning levels are indicated in three types: "Level 0", which indicates that a problem-free process was detected, "Level 1", which indicates that a process suspected of fraudulent activity was detected, and "Level 2", which indicates that fraudulent activity was detected. The version data is data that indicates the version of the analysis result file. The version data is updated every time a value is added, changed, or deleted in the analysis result file.

[0015] The detection unit 133 detects an object from the video data acquired by the acquisition unit 131, and outputs detection result data indicating the frame in which the object was detected, the type of the detected object, the area in which the object was detected, and the reliability of the detection. Specifically, the detection unit 133 inputs each frame of the video data to a trained object detection model, and performs an object detection process for each frame. The object detection model is a model trained to output the coordinates of a bounding box including an object included in the image and the type of object included in each bounding box when an image is input. The object detection model is trained using a learning dataset consisting of input samples, which are frames of video data, and output samples, which are combinations of bounding boxes surrounding objects included in the input samples and the type of object. The type of object to be detected is selected in advance in consideration of the business content, and is likely to be a target of detection. The detection result data indicating the detection result of the object is recorded in the video storage unit 132 in association with the video data of the detection target.

[0016] The setting unit 134 accepts the setting of a business rule indicating the start condition and end condition of the business and the detection condition of the fraudulent activity for each space and business to be monitored. The start condition and the detection condition are represented by the type of the object to be detected and the detection state of the object. The detection state is selected from whether the object is detected, whether the state in which the object is detected continues, and whether the state in which the object is not detected continues. The detection state may include the designation of the area to be detected in the frame of the video data. The end condition may be represented by the detection state of the object, or may be represented by the elapsed time since the start condition is satisfied. The business rule set by the setting unit 134 is recorded in the setting storage unit 135 in association with the name of the business rule.

[0017] The selection unit 136 accepts, for each piece of video data stored in the video storage unit 132, a selection of a business rule to be used for monitoring the video data. The determination unit 137 determines whether each frame of the video data satisfies the detection condition of the business rule based on the analysis result of the video data by the detection unit 133 and the business rule associated with the video data. The determination unit 137 identifies the frame number of the frame that satisfies the detection condition.

[0018] The generating unit 138 generates index data that associates the frame number identified by the determining unit 137 with the detection condition detected in the frame indicated by the frame number. The output unit 139 outputs the analysis result file stored in the video storage unit 132 to the client terminal 15 .

[0019] FIG. 4 is a schematic block diagram showing the configuration of the client terminal 15 according to the first embodiment. The client terminal 15 includes a display control unit 151 , a request unit 152 , a playback unit 153 , and an editing unit 154 . The display control unit 151 controls the display on the display based on information received from the video server 13 . The request unit 152 transmits a request to the video server 13 based on the user's input. The playback unit 153 plays back the video data acquired from the video server 13. The playback unit 153 accepts an index designation based on the index data from the user, thereby designating a playback position of the video data. The editing unit 154 edits the index data of the analysis result file stored in the video server 13. Specifically, the editing unit 154 adds an index indicating the playback position of the video data being played back by the playback unit 153 to the index data.

[0020] How to set up business rules 5 is a flowchart showing a method for setting business rules by the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 transmits a request for rule information to the video server 13 (step S1). When the output unit 139 of the video server 13 receives the request for rule information from the client terminal 15, it transmits to the client terminal 15 the file name of the video data stored in the video storage unit 132, the analysis result file, and the business rule stored in the setting storage unit 135 (step S2).

[0021] The display control unit 151 of the client terminal 15 generates an analysis target selection screen based on the file name of the video data and the analysis result file received from the video server 13, and displays it on the display (step S3). FIG. 6 is a diagram showing an example of the analysis target selection screen according to the first embodiment. The analysis target selection screen displays a video list i01 for selecting an analysis target from a plurality of video data, and a setting start button i02 for proceeding to the setting of business rules for the selected video data. The video list i01 includes a check box i011 for accepting a selection of whether or not to select the video data as an analysis target, a mark i012 indicating the analysis result, and a file name i013. The mark i012 indicates whether or not the analysis of the associated video data has been completed, and if completed, whether or not a fraudulent act or a confirmatory act has been detected. Note that if both a fraudulent act and a confirmatory act have been detected, the mark i012 is displayed in a manner indicating that a fraudulent act has been detected.

[0022] The request unit 152 of the client terminal 15 determines whether the setting start button i02 is pressed by the user (step S4). If the setting start button i02 is not pressed (step S4: NO), the request unit 152 returns the process to step S4 and waits for the setting start button i02 to be pressed. On the other hand, if the setting start button i02 is pressed (step S4: YES), a check box i011 associated with the video data to be analyzed is turned ON on the analysis target selection screen, and when the button i02 is pressed, the display control unit 151 of the client terminal 15 generates a business rule setting screen based on the file name and business rule of the video data received from the video server 13 in step S2, and displays it on the display (step S5). FIG. 7 is a diagram showing an example of a business rule setting screen according to the first embodiment. The business rule setting screen includes a video list i11, a rule list i12, a rule display pane i13, an add rule button i14, and an analysis start button i15.

[0023] The video list i11 displays the file name of the video data related to the selected check box i011. The video list i11 is configured to allow selection of one or more files from among multiple video data. The rule list i12 displays the names of business rules received from the video server 13. A check box is associated with each business rule name, and a selection is accepted as to whether or not to set the business rule to the video data selected in the video list i11. When a business rule in the rule list i12 is clicked, the contents of the business rule are displayed in the rule display pane i13. The Add Rule button i14 accepts the addition of a business rule when pressed. The Start Analysis button i15, when pressed, executes an analysis based on the associated business rule for each video data to be analyzed.

[0024] The request unit 152 of the client terminal 15 determines whether or not the rule addition button i14 has been pressed by a user (step S6). If the rule addition button i14 has been pressed (step S6: YES), the display control unit 151 displays a rule creation screen on the display (step S7). The rule creation screen includes a form for receiving input of a rule name, a start condition, an end condition, a detection condition, and detection accuracy.

[0025] The request unit 152 accepts settings of the type of object to be detected from video data and the detection state as the start condition of the business rule. The type of object is selected from a list of types detectable by the object detection model used by the detection unit 133 of the video server 13. The detection state is selected from either a change from an undetected state to a detected state of the object, or a change from a detected state to an undetected state of the object. In addition, the request unit 152 is capable of accepting settings of a frame area as an option for the detection state. The request unit 152 receives, as an end condition of a business rule, a setting of the elapsed time from the detection state of an object or the point in time when a start condition is satisfied.

[0026] The request unit 152 accepts settings of the type of object to be detected from the video data and the detection state as the detection condition of the business rule. The detection state related to the detection condition is selected from any one of the following: the object has changed from an undetected state to a detected state, the object has changed from a detected state to an undetected state, the object has not changed from an undetected state to a detected state until the end condition is satisfied, and the object has not changed from a detected state to an undetected state until the end condition is satisfied. In addition, the request unit 152 can accept settings of a frame area as an option for the detection state. The request unit 152 accepts an integer greater than 0 and equal to or less than 100 as the detection accuracy of an object in the business rule. The detection accuracy may be expressed by a numerical value between 0 and 1.0. The detection accuracy may be a numerical value input directly or input using a slider. In this case, the input range of the slider may be set to, for example, 70 to 100, allowing the specification of a finer numerical value for the detection accuracy to be accepted.

[0027] When the request unit 152 accepts an input of a business rule from a user, it transmits the input business rule to the video server 13 (step S8). When the setting unit 134 of the video server 13 receives the business rule from the client terminal 15, it associates the received business rule with the business rule name and records it in the setting storage unit 135 (step S9). The video analysis support system 1 returns the process to step S2 and transmits information including the added business rule to the client terminal 15.

[0028] If the Add Rule button i14 has not been pressed on the business rule setting screen (step S6: NO), the request unit 152 determines whether the Start Analysis button i15 has been pressed by the user (step S10). If the Start Analysis button i15 has not been pressed (step S10: NO), the request unit 152 returns the process to step S6 and monitors whether a button has been pressed.

[0029] When the analysis start button i15 is pressed (step S10: YES), the request unit 152 transmits a video analysis request to the video server 13 (step S11). The analysis request includes information indicating the relationship of each video displayed in the video list i11 with the business rule indicated by the check box that is turned ON in the rule list i12 for that video. The selection unit 136 of the video server 13 receives the analysis request and accepts the selection of the business rule to be applied to each video (step S12). The video server 13 analyzes each video stored in the video storage unit 132 according to the selected business rule (step S13).

[0030] FIG. 8 is a flowchart showing an analysis processing method performed by the video server 13 according to the first embodiment. When the video server 13 starts the analysis based on the business rules, it selects each video data recorded in the video storage unit 132 (step S31), and executes the following processes from step S32 to step S39.

[0031] The detection unit 133 of the video server 13 determines whether or not the video storage unit 132 contains detection result data associated with the video data selected in step S31 (step S32). If there is no detection result (step S32: NO), the detection unit 133 uses an object detection model to detect an object from each frame of the video data selected in step S31 (step S33). The detection unit 133 records the detection result data indicating the detection result in the video storage unit 132 in association with the video data of the detection target (step S34). The detection result data includes, for each frame, a frame number, the type of object detected in the frame, the coordinates of a bounding box surrounding the object, and a reliability level, which are associated with each other. The generation unit 138 initializes the index data associated with the video data selected in step S31 (step S35). At this time, the generation unit 138 deletes indexes (indexes with record type A) that were added through analysis based on business rules from the index data, but does not delete indexes (indexes with record type M) that were added manually by the administrator.

[0032] Next, the determination unit 137 selects one by one of the business rules selected in step S10 as the business rule to be applied to the video data selected in step S31 (step S36), and executes the processes in the following steps S37 to S39.

[0033] The determination unit 137 identifies a frame (start frame) that satisfies the start condition indicated by the business rule selected in step S36 and a frame (end frame) that satisfies the end condition indicated by the business rule selected in step S36 based on the analysis result file of the video data selected in step S31 (step S37). The determination unit 137 identifies a frame that satisfies the detection condition indicated by the business rule selected in step S36 and a frame that no longer satisfies the detection condition between the identified start frame and end frame (step S38). If the determination results of whether or not the start condition, end condition, or detection condition are satisfied are different for a predetermined number of frames or less (for example, one frame) among consecutive frames, the determination unit 137 regards the frame as a detection miss or a false detection, and aligns the determination result of the corresponding frame with the determination results of the other frames. For example, if the detection condition is satisfied in 19 frames out of 20 consecutive frames and the detection condition is not satisfied in only one frame, the determination unit 137 treats the one frame as satisfying the detection condition. The number of frames that are regarded as a detection miss or a false detection can be changed by an operation of an administrator.

[0034] For example, when the detection condition indicates that an object has been detected, the determination unit 137 identifies the first frame that indicates that an object has been detected as the frame that satisfies the detection condition. On the other hand, when the detection condition indicates that an object has been detected, and there is no frame in which an object has been detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection condition is no longer satisfied. When the detection condition indicates that the object is no longer detected, the determination unit 137 identifies the first frame that indicates a state in which the object is not detected as a frame that satisfies the detection condition. On the other hand, when the detection condition indicates that the object is no longer detected, and there is no frame in which the object is not detected between the start frame and the end frame, the determination unit 137 identifies the end frame as a frame in which the detection state is no longer satisfied. When the detection condition indicates that the state in which the object is detected continues, the determination unit 137 identifies the first frame in which the object is not detected as the frame in which the detection condition is no longer satisfied. On the other hand, when the detection condition indicates that the state in which the object is detected continues, and there is no frame in which the object is not detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame in which the detection state is satisfied. When the detection condition indicates that the state where the object is not detected continues, the determination unit 137 identifies the first frame showing the state where the object is detected as the frame where the detection condition is no longer satisfied. On the other hand, when the detection condition indicates that the state where the object is not detected continues and there is no frame where the object is detected between the start frame and the end frame, the determination unit 137 identifies the end frame as the frame where the detection state is satisfied.

[0035] The generating unit 138 adds indexes representing the frames identified in steps S37 and S38 to the index data initialized in step S35 (step S39). As shown in FIG. 3, each index indicates a frame number, a message representing the corresponding state, a record type, and a warning level. For example, in a frame that satisfies a detection condition indicating fraudulent activity, the generating unit 138 sets the warning level to level 1 when the reliability of the detected object is lower than a reference value, and sets the warning level to level 2 when the reliability is equal to or higher than the reference value. The reference value may be, for example, a value obtained by adding a predetermined value to the detection accuracy set in the business rule.

[0036] This enables the video server 13 to generate index data in which indexes of frames that correspond to the business rule selected in step S12 are recorded for each video data.

[0037] "Method for playing video data" 9 is a flowchart showing a method for playing video data by the video analysis support system 1 according to the first embodiment. The administrator operates the client terminal 15 to access the video server 13. The request unit 152 of the client terminal 15 transmits a request for video information to the video server 13 (step S51). Upon receiving the request for video information from the client terminal 15, the output unit 139 of the video server 13 transmits to the client terminal 15 the file name of the video data stored in the video storage unit 132 and index data associated with the video data (step S52).

[0038] The display control unit 151 of the client terminal 15 generates a video selection screen based on the file name and index data of the video data received from the video server 13, and displays it on the display (step S53). FIG. 10 is a diagram showing an example of the video selection screen according to the first embodiment. On the video selection screen, a file name i31 and a mark i32 indicating an analysis result are displayed for each video data. The video selection screen is configured to allow selection of one of a plurality of video data. When one of the file names i31 displayed on the video selection screen is selected, a playback screen of the selected video data is displayed. The mark i32 indicates whether or not the analysis of the associated video data has been completed, and if completed, whether or not a fraudulent act has been detected. That is, when there is no index data received from the video server 13, the display control unit 151 attaches a mark i32 indicating that the analysis of the video data has not been completed. When index data received from the video server 13 exists, and when the analysis result file contains one or more pieces of warning level 1 or 2, the display control unit 151 attaches a mark i32 indicating that fraudulent activity has been detected. When index data received from the video server 13 exists, and when all warning levels are 0, the display control unit 151 attaches a mark i32 indicating that fraudulent activity has not been detected.

[0039] In other embodiments, the display control unit 151 may display the mark i32 indicating whether or not there is fraudulent activity separately for cases where there is obvious fraud (when there is something with a warning level of 2) and cases where there is suspicion and needs to be checked (when there is nothing with a warning level of 2 and something with a warning level of 1).

[0040] When the request unit 152 of the client terminal 15 accepts the selection of one video data item from the user, it transmits a request for the selected video data item to the video server 13 (step S54). The acquisition unit 131 of the video server 13 acquires the video data item from the video storage unit 132, and transmits it to the client terminal 15 (step S55).

[0041] The playback unit 153 generates a playback screen for the video data and displays it on the display (step S56). Fig. 9 is a diagram showing an example of the playback screen according to the first embodiment. On the playback screen, a playback pane i51 for the video data, an index pane i52, a control pane i53, and an index edit button i54 are displayed.

[0042] The playback pane i51 displays the video data distributed from the video server 13. When an object is detected in the displayed frame, the playback pane i51 displays a bounding box surrounding the object, and the type and reliability of the object. The bounding box can be displayed / hidden by an administrator's operation. The index pane i52 displays a list of indexes indicated by index data associated with the video data displayed in the playback pane i51. In the index pane i52, the index is represented by a combination of a playback position and a message. In addition, in the index pane i52, the index is represented by a different color depending on the warning level. This allows the administrator to easily recognize a portion of the video data where there is a high possibility of fraudulent activity or a portion where fraudulent activity is suspected. The control pane i53 displays buttons for controlling the playback of the video data displayed in the playback pane i51. Specifically, the control pane i53 displays a seek bar indicating the playback position, a back skip button for moving the playback position back a predetermined time, a stop button for stopping playback, a play button for starting playback, a skip button for moving the playback position forward a predetermined time, and the like. The index edit button i54 is used to edit the index data by pressing it.

[0043] The playback unit 153 receives the video data selected in step S54 from the video server 13 and plays it in the playback pane i51 (step S57). The playback unit 153 determines whether or not any one of the indexes displayed in the index pane i52 has been pressed (step S58). If an index has been pressed (step S58: YES), the playback unit 153 moves the playback position of the video data being played to the playback position associated with the pressed index (step S59). This allows the administrator to easily check noteworthy parts of the video data.

[0044] If the index has not been pressed (step S58: NO), the editing unit 154 judges whether or not the index edit button i54 has been pressed (step S60). If the index edit button i54 has been pressed (step S60: YES), the playback unit 153 stops the playback of the video data displayed in the playback pane i51 at the current playback position. When the editing unit 154 accepts input of a message from the administrator, it transmits an index addition request including the current playback position and the input message to the video server 13 (step S61). The generation unit 138 of the video server 13 updates the index data based on the received index addition request (step S62). This allows the administrator to supplement information such as information that has been omitted from the analysis by the video server 13.

[0045] Furthermore, when the index edit button i54 is pressed in a frame in which an index already exists, the editing unit 154 accepts a change or deletion of the contents of the index. For example, when the administrator visually checks a frame in which the warning level is 1, i.e., which indicates a suspected fraudulent activity, and determines whether or not the activity detected in the frame is a fraudulent activity, the administrator may edit the index in the following procedure. The administrator presses the index to be edited, moves the playback position, and then presses the index edit button i54. This causes the editing unit 154 to accept editing of the pressed index. If a fraudulent activity is confirmed, the administrator changes the warning level of the index to 2, and if a fraudulent activity is confirmed, the administrator changes the warning level of the index to 0.

[0046] The request unit 152 determines whether the playback screen has been closed (step S63). If the playback screen has not been closed (step S63: NO), the process returns to step S55 and playback of the video data continues. On the other hand, if the playback screen has been closed (step S63: YES), the video analysis support system 1 ends the playback process of the video data.

[0047] This allows the administrator to efficiently monitor operations by playing back video data using the index data. Also, by setting indexes for points that require attention while the video data is being played back, the administrator can refer to the indexes when later referencing the video data.

[0048] 《Learning function from verified data》 The administrator can determine whether a frame indicated by an index with a warning level of 1 indicates fraudulent activity, and rewrite the warning level based on the result of the determination. The video server 13 may change the business rule based on the warning level rewritten by the administrator. For example, the video server 13 may change the start condition, end condition, detection condition, and detection accuracy based on the business rule. For example, in a business rule in which a detection accuracy of 95% or more is determined to be warning level 2, a detection accuracy of 90% or more but less than 95% is determined to be warning level 1, and a detection accuracy of less than 90% is determined to be warning level 0, if the frequency of rewriting from warning level 1 to warning level 2 is high, the video server 13 may change the threshold value that distinguishes between warning level 1 and warning level 2 to a value lower than 90%, for example. Also, for example, in a business rule in which the end condition is 3 seconds after the expiration of the start condition, if the detection condition is related to fraudulent activity and the frequency of rewriting from warning level 1 to warning level 2 is high, the time related to the end condition may be extended. Also, for example, if the object related to the detection condition is not related to fraudulent activity and the frequency of rewriting from warning level 1 to warning level 2 is high, the time related to the termination condition may be shortened.

[0049] Example of business rule settings Here, an example of a business rule used in analyzing video data will be described. For example, a business rule regarding cabinet operation can be set as a business rule. If the business rule regarding cabinet operation specifies that the cabinet should not be left open, the following business rule can be set: The start condition of the business rule related to cabinet operation is that an object of the type "open cabinet" is detected. The end condition of the business rule related to cabinet operation is that an object of the type "open cabinet" is no longer detected. The detection condition of the business rule related to cabinet operation is that an object of the type "person" is no longer detected. This allows the video server 13 to monitor whether or not the worker has left the cabinet open during cabinet operation.

[0050] For example, a business rule regarding document transfer can be set as a business rule. If the business rule regarding document transfer specifies that documents submitted by customers should be received and stored, that documents submitted to customers should be confirmed to be stored, and that lost documents should be discovered, the following business rule can be set. The start condition of the business rule related to the transfer of documents is that two objects of the type "person" and an object of the type "document" are detected. The end condition of the business rule related to the transfer of documents is that the two objects of the type "person" are no longer detected. The detection condition of the business rule related to the transfer of documents is that the object of the type "document" continues to be not detected, and that the object of the type "document" is no longer detected in a position other than the area where it should be stored. Note that the type of object in this business rule is "document", but in other business rules, conditions for other types such as "banknote", "coin", and "card" may be set. In other business rules, the type of document may be specified in more detail than "document" (contract, identification document, pamphlet, etc.). This allows the video image server 13 to monitor whether the document has been properly exchanged.

[0051] For example, a business rule regarding entry and exit into the vault can be set as a business rule. If the business rule regarding entry and exit into the vault stipulates that two or more people are to enter and exit the vault, the following business rule can be set. The start condition of the business rule related to entry into the vault is that an object of the type "person entering" has been detected. An "entering person" is, for example, a person facing inward into the vault. The end condition of the business rule related to entry into the vault is that a certain amount of time has elapsed since the start condition has been satisfied. The detection condition of the business rule related to entry into the vault is that a state in which an object of the type "person entering", different from the object of the start condition, has not been detected continues. The start condition of the business rule related to leaving the vault is that an object of the type "person leaving the vault" has been detected. A "person leaving the vault" is, for example, a person facing away from the vault. The end condition of the business rule related to leaving the vault is that a certain amount of time has elapsed since the start condition has been satisfied. The detection condition of the business rule related to leaving the vault is that a state in which an object of the type "person leaving the vault", different from the object of the start condition, has not been detected continues. This allows the video server 13 to monitor whether or not two or more people are entering or leaving the safe room. In addition, in the business rule for entering and leaving the vault, the detection condition may include that at least one of the "persons" is an object of the type "administrator." In other words, the detection condition for the business rule for entering and leaving the vault may include a state in which an object of the type "administrator" is not detected for a long period of time.

[0052] For example, a business rule for operating a cash register can be set as a business rule. If it is determined that a general worker and a manager are to work together to access the inside of the cash register, the following business rule can be set as the business rule for operating the cash register. The start condition of the business rule related to the operation of the teller machine is that an object of the type "person" is detected near the teller machine. The end condition of the business rule related to the operation of the teller machine is that an object of the type "person" is no longer detected near the teller machine. The detection condition of the business rule related to the operation of the teller machine is that a state in which an object of the type "person" different from the object of the start condition is not detected continues. The detection condition of the business rule related to the operation of the teller machine may be that a state in which an object of the type "manager" is not detected continues. This enables the video server 13 to monitor whether or not the cash dispenser operation is being performed in the presence of a manager. The above business rules can be applied to devices such as cash management devices, valuables management devices, and key management devices in addition to teller machines.

[0053] For example, a business rule can be set regarding the posting of discarded documents in a document processing device such as a shredder or document dissolving machine. If it is determined that the task of posting discarded documents in a document processing device is to be performed by a general worker and a manager, the following business rule can be set as the business rule regarding the posting of discarded documents. The start condition of the business rule relating to the deposit of documents to be discarded is that an object of the type "person" is detected near the document processing device. The end condition of the business rule relating to the deposit of documents to be discarded is that an object of the type "person" is no longer detected near the document processing device. The detection condition of the business rule relating to the deposit of documents to be discarded is that a state in which an object of the type "person" different from the object of the start condition is not detected continues. The detection condition of the business rule relating to the deposit of documents to be discarded may also be a state in which an object of the type "administrator" is not detected continues. This allows the video image server 13 to monitor whether or not the disposal document is posted in the presence of an administrator.

[0054] Furthermore, if it is stipulated that documents to be discarded be inspected before being placed in the document processing device, the following business rule can be set. In this case, the start condition of the business rule related to the deposit of a document to be discarded is that an object of the type "person" is detected near the document processing device. The end condition of the business rule related to the deposit of a document to be discarded is that an object of the type "person" is no longer detected near the document processing device. The detection condition of the business rule related to the deposit of a document to be discarded is that the state in which "documents" are detected continues for a certain period of time before the object of the start condition, that is, before the "documents" are deposited in the document processing device, the object of the type "documents" is no longer detected in the position where the objects of the type "documents" should be stored. This allows the video server 13 to monitor whether a "person" is checking the documents before depositing them for disposal.

[0055] For example, a business rule regarding postmarks on documents can be set as a business rule. When a business rule for documents is performed to postmark a specific document as proof of receipt of the document, the following business rule can be set. The start condition for a business rule related to postmarks on documents is that an object of the type "person" and an object of the type "document" are detected. The end condition for a business rule related to postmarks on documents is that objects of the type "person" and objects of the type "document" are no longer detected. The detection condition for a business rule related to postmarks on documents is that an object of the type "postmark" is detected superimposed on "document", which is the object of the start condition. This allows the video image server 13 to monitor whether or not a postmark has been placed on a document.

[0056] For example, as the business rules, it is possible to set business rules regarding handover at a security center, a nurse center, etc. As the business rules for handover, when performing a task of handing over specific documents together with a roll call, the following business rules can be set. The start condition of the business rule related to handover is that two objects of the type "person" and an object of the type "document" are detected. The end condition of the business rule related to handover is that the two objects of the type "person" are no longer detected. The detection conditions of the business rule related to handover are that a state in which an object of the type "person pointing" is not detected continues, and that an object of the type "document" is no longer detected in a position that is not in an area where it should be stored. This allows the video server 13 to monitor whether the handover has been performed appropriately.

[0057] Actions and Effects Thus, according to the first embodiment, the video analysis support system 1 detects frames indicating a state in which a task was not performed correctly from video data captured by the surveillance camera 11 based on the business rules, and generates index data indicating the frames. This allows the manager to check the video data based on the index data. In other words, the video analysis support system 1 can support the manager to easily check whether the task is being performed correctly from the video captured by the surveillance camera 11.

[0058] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel. The video server 13 according to the above-described embodiment may be configured by a single computer, or the configuration of the video server 13 may be divided and arranged among a plurality of computers, and the plurality of computers may cooperate with each other to function as the video server 13. In other embodiments, the video server 13 and the client terminal 15 may be configured by a single computer.

[0059] The video server 13 according to the embodiment described above sets the detection accuracy for determining the presence or absence of detection for each business rule, but is not limited to this. For example, in other embodiments, the detection accuracy may be set for each type of object, or the detection accuracy may be a fixed value.

[0060] The video server 13 according to the above embodiment records the detection result data in the setting storage unit 135 when the detection unit 133 performs detection, and skips analysis of video data for which detection result data has already been generated. In this way, when the detection accuracy setting in the business rules is changed after detection by the detection unit 133, the time required for generating the analysis result file can be shortened by using the detection result data that has already been generated. On the other hand, other embodiments are not limited to this. For example, the video server 13 according to other embodiments may perform detection by the detection unit 133 every time an analysis request is received. When the detection unit 133 performs detection each time in this manner, management of the analysis result file in the video server 13 can be facilitated.

[0061] The start condition and end condition of the business rule according to the embodiment described above are determined based on the detection result by the detection unit 133, but are not limited to this. For example, the start condition and end condition of the business rule according to another embodiment may be determined based on log data of a device installed in the monitored space. For example, the start condition may be that a predetermined type of cash dispenser operation has been performed on a cash dispenser installed in the monitored space, and a frame that satisfies the start condition may be identified from the log data of the cash dispenser.

[0062] The detection condition of the business rule according to the above embodiment specifies a fraudulent act, i.e., a state in which the business is not executed correctly, but is not limited thereto. For example, the detection condition according to another embodiment may specify a state in which the business is executed correctly.

[0063] Computer Configuration FIG. 10 is a schematic block diagram illustrating a computer configuration according to at least one embodiment. The computer 90 comprises a processor 91, a main memory 93, a storage 95, and an interface 97. The above-mentioned video server 13 and client terminal 15 are each implemented in a computer 90. The operations of each of the above-mentioned processing units are stored in the form of a program in a storage 95. The processor 91 reads the program from the storage 95, loads it in the main memory 93, and executes the above-mentioned processes in accordance with the program. The processor 91 also secures storage areas in the main memory 93 corresponding to each of the above-mentioned storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0064] The program may be for realizing a part of the functions to be performed by the computer 90. For example, the program may be for realizing the functions by combining with other programs already stored in the storage or by combining with other programs implemented in other devices. In another embodiment, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to the above configuration or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions to be realized by the processor 91 may be realized by the integrated circuit. Such an integrated circuit is also included in an example of a processor.

[0065] Examples of the storage 95 include a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, etc. The storage 95 may be an internal medium directly connected to the bus of the computer 90, or an external medium connected to the computer 90 via the interface 97 or a communication line. In addition, when the program is distributed to the computer 90 via a communication line, the computer 90 that receives the program may load the program into the main memory 93 and execute the above-mentioned processing. In at least one embodiment, the storage 95 is a non-transitory tangible storage medium.

[0066] The program may be for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with another program already stored in the storage 95. [Explanation of symbols]

[0067] 1...Video analysis support system 11...Surveillance camera 13...Video server 131...Acquisition unit 132...Video storage unit 133...Detection unit 134...Setting unit 135...Setting storage unit 136...Selection unit 137...Determination unit 138...Generation unit 139...Output unit 15...Client terminal 151...Display control unit 152...Request unit 153...Playback unit 154...Editing unit 90...Computer 91...Processor 93...Main memory 95...Storage 97...Interface

Claims

1. An acquisition unit that acquires video data captured by an imaging device that captures an image of a space where a business is performed; a setting unit that receives a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data; a determination unit that determines whether or not a frame that satisfies the detection condition is present from the video data; a generating unit for generating index data indicating a frame defined in the detection condition; Equipped with When the detection results of a predetermined number or less of consecutive frames are different, the determination unit aligns the determination result of the corresponding frame with the determination results of the other frames. Video image analysis support device.

2. The other frames are frames that satisfy the detection condition among the consecutive frames. The video analysis support device according to claim 1 .

3. The other frames are the frames before and after the corresponding frame among the consecutive frames. The video analysis support device according to claim 1 .

4. The video analysis support device according to claim 1 , wherein the predetermined number is changeable.

5. On the computer, A step of acquiring moving image data captured by an imaging device that captures an image of a space where a business is performed; receiving a setting of a business rule indicating a detection condition that specifies a state in which the business is correctly executed or a state in which the business is not correctly executed in the frame of the video data; determining whether or not a frame that satisfies the detection condition is present from the video data; generating index data indicating a frame defined in the detection condition; Run the command, In the step of determining whether or not a frame that satisfies the detection condition exists from the video image data, if the detection results of a predetermined number or less of consecutive frames are different, the determination result of the corresponding frame is made uniform with the determination results of the other frames. program.

Images