Specific behavior detection device, method, and program

Abstract

An embodiment of the present invention acquires video data obtained by imaging a target area, and detects, from a frame in the acquired video data, a partial area video including at least an object that is a detection target. Then, for each first section including a plurality of said frames, the embodiment extracts feature quantity information representing a feature of behavior of the object from each of a plurality of said partial area videos detected in the first section, and structures the extracted plurality of sets of feature quantity information over a second section to generate first behavioral feature information. The embodiment then calculates a cost reflecting the degree of similarity between the generated first behavioral feature information and each of a plurality of sets of second behavioral feature information relating to the object and prepared in advance, detects, as search behavior information, a set of second behavioral feature information for which the calculated cost satisfies a preset condition, and outputs the detected search behavior information.

Description

[Technical field]

[0001] One aspect of the present invention relates to a specific behavior detection device, method, and program for detecting a specific behavior of a person from media data such as video data. [Background technology]

[0002] In recent years, with the spread of high-definition cameras, various technologies have been proposed to analyze human behavior based on captured images. For example, there are technologies that detect criminal behavior based on surveillance camera images and abnormal behavior at construction sites.

[0003] Incidentally, in order to detect human behavior with high accuracy, it is necessary to observe a large amount of video. In such a case, since manual detection requires time and human costs, technology that uses algorithms to automatically detect abnormal behavior has been researched. For example, there is a technology that detects abnormal behavior with high accuracy by clustering videos using a neural network. However, this type of technology currently proposed does not provide sufficient discrimination performance when detecting rare abnormal behavior from a large amount of normal behavior. In addition, there are many scenes in which it is unclear what kind of behavior the abnormal behavior itself is, and there are cases in which it is difficult to label normal and abnormal.

[0004] In view of this, as described in Non-Patent Document 1, for example, a technique has been proposed in which a user is asked to specify an abnormal behavior as a query, and the specified query is used to detect the abnormal behavior. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] D. Dwibedi et al. “Temporal Cycle Consistency Learning.” CVPR2019. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Non-Patent Document 1 detects abnormal behavior by considering all frames of the video, that is, by targeting all the features extracted from each frame of the video without considering the time characteristics of the person's movements. Therefore, when the length of the query video is short and the video spans a long period of time, the calculation cost becomes very high.

[0007] This invention has been made in light of the above-mentioned circumstances, and aims to provide a technology that enables the detection of specific behaviors that take into account the temporal structure of a person's motion characteristics while keeping computational costs down. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of the specific behavior detection device or method according to the present invention includes a first processing unit or process for acquiring video data capturing an image of a target area, a second processing unit or process for detecting a partial area image including at least an object to be detected from a frame of the acquired video data, and feature amount information representing a feature of the movement of the object from a plurality of the partial area images detected in the first section for each first section including a plurality of the frames. a graph in which each of the feature values ​​is assigned to a number of nodes corresponding to the second interval, thereby structuring the plurality of feature value information; a third processing unit or step of generating first behavior characteristic information; a fourth processing unit or step of calculating a cost reflecting a degree of similarity between the generated first behavior characteristic information and each of a plurality of second behavior characteristic information related to the object prepared in advance, and detecting the second behavior characteristic information for which the calculated cost satisfies a preset condition as search behavior information; and a fifth processing unit of outputting the detected search behavior information. or process It is equipped with the following. The fourth processing unit or process executes a process of mutually replacing each first intermediate node and second intermediate node between the first behavior characteristic information and each of a plurality of second behavior characteristic information, a process of calculating first similarity information representing a sum of similarities between the replaced second or first intermediate node and each other node of the first or second behavior characteristic information, a process of calculating second similarity information representing a sum of similarities between the second or first intermediate node of the second or first behavior characteristic information and each other node of the second or first behavior characteristic information, a process of calculating a difference between the first similarity information and the second similarity information as a replacement cost, and a process of detecting the second behavior characteristic information, for which the replacement cost satisfies a preset condition, as the search behavior information.

[0009] According to one aspect of the present invention, for example, a search process for a person's behavioral characteristic information is performed by comparing a first behavioral characteristic information structured by a section of a plurality of frames with a plurality of second behavioral characteristic information prepared in advance. Therefore, the time required for the search process can be significantly reduced compared to a case where the behavioral characteristic information of a person shown in each frame of video data is compared with the behavioral characteristic information of the search target, and the processing load of the specific behavior detection device required for the search process can be reduced. Effect of the Invention

[0010] That is, according to one aspect of the present invention, it is possible to provide a technology that enables detection of a specific behavior taking into account the temporal structure of a person's motion characteristics while keeping calculation costs down. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of a specific behavior detection device according to a first embodiment of the present invention, together with the configuration of its peripheral parts. [Diagram 2] FIG. 2 is a block diagram showing an example of a software configuration of the specific behavior detection device according to the first embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart showing an example of a processing procedure and processing contents of a specific behavior detection process executed by the control unit of the specific behavior detection device shown in FIG. [Figure 4] FIG. 4 is a flowchart showing an example of a processing procedure and processing contents of the behavior feature extraction processing in the specific behavior detection processing shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing an example of a processing procedure and processing contents of the behavior search processing in the specific behavior detection processing shown in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of a processing procedure and processing contents of the search behavior ranking processing in the specific behavior detection processing shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [First embodiment] (Configuration example) FIG. 1 is a block diagram showing an example of the hardware configuration of a specific behavior detection device according to a first embodiment of the present invention, together with its peripheral devices, and FIG. 2 is a block diagram showing an example of the software configuration of the specific behavior detection device.

[0014] The specific behavior detection device BD is composed of an information processing device such as a server computer or a personal computer, and a camera CM and a terminal MT are connected to this specific behavior detection device BD via a signal cable or a network (not shown).

[0015] The camera CM is installed, for example, on a ceiling or wall surface capable of capturing an image of the monitored area, captures the entire body or part of a person to be detected who is present in or has entered the monitored area, along with the surrounding area, and transmits the time-series video data to the specific behavior detection device BD.

[0016] The video data captured by the camera CM may be directly transmitted from the camera CM to the specific behavior detection device BD, or may be temporarily stored in a video database (not shown) and then transmitted to the specific behavior detection device BD. The number of cameras CM is not limited to one, and may be multiple.

[0017] The terminal MT is used by, for example, a system administrator or a monitor who monitors a person's specific behavior, and is configured as an information processing terminal such as a personal computer. The terminal MT has a function of receiving and displaying detection information of specific behavior output from the specific behavior detection device BD. Furthermore, when the specific behavior detection device BD has a machine learning model, the terminal MT may have a function of inputting learning data required for learning the machine learning model to the specific behavior detection device BD.

[0018] The specific behavior detection device BD includes a control unit 1 using hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU), and is connected to the control unit 1 via a bus 5 with a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, the interface will be abbreviated as I / F) unit 4. The control unit 1 may be configured using a programmable logic device (PLD), a field programmable gate array (FPGA), or the like.

[0019] The input / output I / F unit 4 has a communication interface function, and transmits and receives video data and various input / output data between the camera CM and the terminal MT via a signal cable or a network.

[0020] The program storage unit 2 is configured by combining a non-volatile memory such as a solid state drive (SSD) as a storage medium that can be written and read at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs required for executing various control processes according to the first embodiment, in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

[0021] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and is equipped with a video data storage unit 31 and a behavioral characteristic information storage unit 32 as main storage units required for implementing the first embodiment.

[0022] The video data storage unit 31 is used to temporarily store the time-series video data transmitted from the camera CM for specific behavior detection processing.

[0023] The behavior characteristic information storage unit 32 stores information representing characteristics of a plurality of specific behaviors that a person to be detected can take as a reference behavior characteristic information group.

[0024] The control unit 1 includes, as processing functions necessary for implementing the first embodiment, a video data acquisition processing unit 11, a person-related area video detection processing unit 12, a behavior feature extraction processing unit 13, a behavior search processing unit 14, a search behavior ranking processing unit 15, and a behavior evaluation processing unit 16. All of the above processing units 11 to 16 are realized by causing a hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

[0025] The video data acquisition processing unit 11 acquires time-series video data output from the camera CM via the input / output I / F unit 4, and performs processing to temporarily store each of the acquired video data in the video data storage unit 31.

[0026] The person-related area video detection processing unit 12 reads video data from the video data storage unit 31 frame by frame, and inputs the frame images of this video data into a pre-trained object detection model, and outputs person-related area video information representing the class and position of the person appearing in the frame image using the machine learning model.

[0027] The behavior feature extraction processing unit 13 inputs the person-related area video information output from the person-related area video detection processing unit 12 to a previously trained action recognition model. Then, using the action recognition model, for each segment in which a plurality of frames are grouped together as one unit, a feature vector relating to the movement of a person corresponding to a group of person-related area videos included in the segment is extracted. Then, the extracted group of behavior vectors of the person is output as time-structured behavior feature information. This person feature information extraction process will be described in more detail in an operation example.

[0028] The behavior search processing unit 14 receives the extracted behavior feature information from the behavior feature extraction processing unit 13, and reads a group of reference behavior feature information associated with the person to be detected from the behavior feature information storage unit 32. The behavior search processing unit 14 then compares the extracted behavior feature information with each of the reference behavior feature information included in the read reference behavior feature information group to calculate a replacement cost of the intermediate node, and outputs the reference behavior feature information whose calculated replacement cost satisfies a predetermined condition by including it in the search behavior information. This behavior search processing will also be described in more detail in an operation example.

[0029] The search behavior ranking processing unit 15 inputs the search behavior information output from the behavior search processing unit 14 to a ranking model trained in advance, and outputs the search behavior information ranked by the ranking model, for example, in order of substitution cost.

[0030] The behavior evaluation processing unit 16 calculates an evaluation value of the search results based on the search behavior information ranked by the search behavior ranking processing unit 15, and performs processing to update the reference behavior characteristic information stored in the behavior characteristic information storage unit 32 based on this evaluation value and the evaluation value input by the user.

[0031] The machine learning models used in the person-related area video detection processing unit 12, the behavioral feature extraction processing unit 13, the behavioral search processing unit 14, and the search behavior ranking processing unit 15 are, for example, configured using a convolutional neural network, but the type of neural network can be appropriately selected and used.

[0032] (Example of operation) Next, an example of the operation of the specific behavior detection device BD configured as above will be described. The following explanation will be given assuming that the machine learning models used in the person-related area video detection processing unit 12, behavior feature extraction processing unit 13, behavior search processing unit 14, and search behavior ranking processing unit 15 have been trained in advance.

[0033] FIG. 3 is a flowchart showing an example of the overall processing procedure and processing contents of the specific behavior detection processing executed by the control unit 1 of the specific behavior detection device BD.

[0034] (1) Acquisition of video data In step S10, the control unit 1 of the specific behavior detection device BD, under the control of the video data acquisition processing unit 11, acquires time-series video data obtained by capturing an image of the monitored area by the camera CM via the input / output I / F unit 4. Then, the acquired video data is temporarily stored in the video data storage unit 31.

[0035] (2) Detection of human-related areas in video When the video data for a certain period of time is acquired, the control unit 1 of the specific behavior detection device BD, under the control of the person-related area video detection processing unit 12, performs processing to detect video of a person-related area from the video data in step S20 as follows.

[0036] That is, the person-related area video detection processing unit 12 reads the video data from the video data storage unit 31 frame by frame, and inputs the frame images of the read video data to the object detection model. Then, the object detection model obtains information indicating the class and position of the person appearing in the frame image.

[0037] For example, the person-related area video detection processing unit 12 detects video sections in which people appear from each frame image of the video data, and cuts out the areas in which people appear for each person using a person tracking method. Then, when the same person is detected continuously for a certain period of time, it outputs person-related area video information with a person ID that identifies the person.

[0038] (3) Extraction of behavioral characteristics Next, in step S30, the control unit 1 of the specific behavior detection device BD, under the control of the behavior characteristic extraction processing unit 13, extracts characteristics of the person's behavior from the person-related area video information as follows.

[0039] FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the behavioral characteristic extraction processing executed by the behavioral characteristic extraction processing unit 13.

[0040] That is, first, in step S31, the action feature extraction processing unit 13 receives person-related area video information from the person-related area video detection processing unit 12. Then, in step S32, the received person-related area video information is input to the trained action recognition model. The action recognition model groups multiple frames of the person-related area video information into one segment, and extracts, for each segment, a feature vector that is a feature amount related to the movement of a person corresponding to a group of person-related area videos included in that segment.

[0041] For example, the behavior characteristic extraction processing unit 13 groups 32 frames into one segment and extracts a characteristic vector from the person-related area video group of the 32 frames. As a result, a characteristic vector corresponding to the person-related area video group of one segment related to the same person is extracted. That is, this process extracts information representing the characteristics of the movement of each person in a 32-frame section.

[0042] Next, in step S33, the behavioral feature extraction processing unit 13 temporally structures the group of feature vectors extracted for each person to generate behavioral feature information. Here, structuring means constructing a graph U in a certain interval T for a group of feature vectors (u_1, u_2, ..., u_N) of a certain person. Here, T corresponds to the number of nodes in the graph, and one node corresponds to a feature vector u_i. Note that a complete graph is taken as an example of the graph here, but links may be reduced as necessary. However, the number of nodes T is determined in advance.

[0043] The behavior feature extraction processing unit 13 then outputs the time-structured behavior feature information U to the behavior search processing unit 14 in step S34.

[0044] (4) Generation of search behavior information Next, in step S40, the control unit 1 of the specific behavior detection device BD performs processing for searching for a person's behavior under the control of the behavior search processing unit 14 as follows.

[0045] FIG. 5 is a flowchart showing an example of the processing procedure and processing contents of the behavior search processing executed by the behavior search processing unit 14.

[0046] That is, first, in step S41, the behavior search processing unit 14 receives behavior characteristic information from the behavior characteristic extraction processing unit 13, and reads a group of reference behavior characteristic information corresponding to the same person from the behavior characteristic information storage unit 32. Then, in step S42, the behavior search processing unit 14 selects one piece of reference behavior characteristic information from the group of reference behavior characteristic information, and compares the selected reference behavior characteristic information with the extracted behavior characteristic information in step S43. Then, the substitution cost of the intermediate node between them is evaluated.

[0047] For example, assuming that the extracted behavior feature information is U and the selected reference behavior feature information is V, the behavior search processing unit 14 replaces the intermediate nodes u_{T / 2} and v_{T / 2}. Then, using the intermediate node as a base point, it calculates the cosine similarity between v_{T / 2} and each node u_j excluding u_{T / 2}, and calculates the sum C_{uv} of the cosine similarities. Meanwhile, it calculates the cosine similarity between v_{T / 2} and each node v_j excluding v_{i / 2} in advance, and holds the sum C_v. Then, the behavior search processing unit 14 calculates the difference between the sum C_{uv} and C_v, and sets the calculated difference value as the replacement cost.

[0048] Note that other indices may be used as long as they can determine the similarity of graph structures, such as a comparison between C_{vu} and C_u, which are the inverse patterns of the above substitution costs. Also, the cosine similarity s between u_{T / 2} and v_{T / 2} may be combined. This method makes it possible to reduce calculation costs by comparing only graphs with high cosine similarity s without performing calculations on the entire graph.

[0049] The replacement cost is not limited to T / 2, but may be calculated by taking the sum or weighted sum of nodes at positions T / 2+1 and T / 2-1 as base points, for example.

[0050] After the substitution cost of the intermediate node is calculated, the behavior search processing unit 14 compares the substitution cost with a preset threshold TH in step S44 to determine whether the substitution cost is smaller than the threshold TH. If the result of this determination is that the substitution cost is smaller than the threshold TH, in step S45, the reference behavior feature information V used for comparison is added to the search behavior information.

[0051] The behavior search processing unit 14 then determines in step S46 whether the search end condition for the reference behavior characteristic information is satisfied. For example, it determines whether the comparison process with all reference behavior characteristic information is completed. If the result of this determination is that unselected reference behavior characteristic information remains, the process returns to step S43 to select the next reference behavior characteristic information, and the selected reference behavior characteristic information is subjected to the processes in steps S43 to S46.

[0052] Thereafter, the behavior search processing unit 14 repeats the process of steps S43 to S46 for each unselected reference behavior characteristic information. Then, when the process is completed for all the reference behavior characteristic information, the behavior search processing unit 14 proceeds to step S47 and outputs the finally obtained search behavior information to the search behavior ranking processing unit 15.

[0053] Prior to the above-mentioned search process for behavioral feature information, it is advisable to cluster v_{T / 2} for the reference behavioral feature information group and construct a centroid c_k for v. In this way, it becomes possible to perform hierarchical search processing, such as searching for the centroid c_k and then searching for each reference behavioral feature information v_i.

[0054] The cost of the centroid c_k is set to, for example, the average of the costs of the reference behavior feature information v_i belonging to the centroid c_k. In this case, the centroid c_k is compared with the cost C_{cu} when it is replaced with the intermediate node u_{T / 2} of the extracted behavior feature information. If the cost compared with the centroid c_k is greater than the threshold as a result of this comparison, the comparison process with other nodes is omitted and the search process ends.

[0055] (5) Ranking of search behavior Next, in step S50, the control unit 1 of the specific behavior detection device BD, under the control of the search behavior ranking processing unit 15, ranks the search behavior information.

[0056] FIG. 6 is a flowchart showing an example of the processing procedure and processing content of the search behavior ranking processing executed by the search behavior ranking processing unit 15.

[0057] That is, first, in step S51, the search behavior ranking processing unit 15 receives search behavior information from the behavior search processing unit 14. Then, in step S52, the received search behavior information is input to a ranking model. Then, in step S53, the search behavior information ranked by the ranking model, for example, in order of substitution cost, is output, and in step S54, the ranked search behavior information is output from the input / output I / F unit 4 to the terminal MT.

[0058] The ranking model may be constructed in any manner as long as it determines the order in which search results are displayed. For example, it may be constructed by arranging cost differences in ascending order, or may be constructed to output ranking evaluation values ​​using ranking learning or the like.

[0059] The search behavior ranking processing unit 15 provides the ranked search behavior information to the behavior evaluation processing unit 16. The behavior evaluation processing unit 16 calculates an evaluation value of the search results based on the ranked search behavior information, and updates the reference behavior characteristic information stored in the behavior characteristic information storage unit 32 based on the evaluation value.

[0060] (Action and effect) As described above, in the first embodiment, first, based on the image of the area in which the same person is detected for each frame of the video data, the behavior feature extraction processing unit 13 extracts vectors representing the characteristics of the person's movement in a section of multiple frames for each segment consisting of multiple frames, and constructs a graph U corresponding to a predetermined section T for the extracted feature vector group, thereby structuring the feature vector group in time. Next, the behavior search processing unit 14 compares the structured extracted behavior feature information with multiple reference behavior feature information of the corresponding person stored in the behavior feature information storage unit 32, respectively, calculates the substitution cost of the intermediate node between them, and sets the reference behavior feature information with the calculated substitution cost smaller than the threshold TH as search behavior information. Finally, the search behavior ranking processing unit 15 ranks each reference behavior feature information included in the search behavior information, for example, in order of substitution cost, and outputs it.

[0061] Therefore, according to the first embodiment, the search process for the behavior characteristic information of a person is performed by comparing the extracted behavior characteristic information structured by a section of multiple frames with multiple reference behavior characteristic information. Therefore, the time required for the search process can be significantly reduced compared to the case where the behavior characteristic information of the person shown in each frame of the video data is compared with the reference behavior characteristic information, and the processing load of the specific behavior detection device required for the search process can be reduced. In other words, it is possible to reduce the calculation cost required for the search process of the specific behavior detection device.

[0062] [Second embodiment] The second embodiment of the present invention is to detect a specific part that clearly shows the movement of a person and its surrounding area in a person-related area video, and to extract the characteristics of the person's behavior based on the detected person-related area video. Note that, in this embodiment, the drawings used in the first embodiment will be quoted as they are for explanation.

[0063] That is, the person-related area video detection processing unit 12 focuses on, for example, the hand as a part that prominently represents the movement of the person to be detected, and detects a rectangular area including the hand and the object being manipulated by the hand as the person-related area video. The behavior feature extraction processing unit 13 inputs the rectangular area in multiple frames together into the behavior recognition model, thereby obtaining a motion feature vector of the hand and the object being manipulated by the hand, and uses this feature vector to construct behavior feature information.

[0064] When detecting the rectangular area, it is desirable to ensure temporal consistency in determining that the hand is the same person's hand. For example, if a hand position at time t+1 is within a certain distance from the hand position at time t, it is considered to be the hand of the same person. In addition, both a feature vector consisting of the area of ​​the entire person and a feature vector consisting of the area including the hand and the object being manipulated by the hand may be held as one node to form behavior feature information. In this case, the behavior feature information may be subjected to operations such as weighted sum, concatenation, and addition.

[0065] [Third embodiment] In the first embodiment, the behavior search processing unit 14 sets a condition for adding to the search behavior information that the replacement cost is smaller than a threshold value TH. In contrast, in the third embodiment, a search is performed with a condition that the replacement cost is larger than a threshold value, thereby detecting a person's rare abnormal behavior. Note that the third embodiment will also be described by citing the figures used in the first embodiment.

[0066] For example, when the extracted behavior feature information is U and the selected reference behavior feature information is V, the behavior search processing unit 14 replaces the intermediate nodes u_{T / 2} and v_{T / 2}. Then, using this intermediate node as a base point, it calculates the cosine similarity between v_{T / 2} and each node u_j excluding u_{T / 2}, and calculates the sum C_{uv} of these cosine similarities. Meanwhile, it calculates the cosine similarity between v_{T / 2} and each node v_j excluding v_{i / 2} in advance, and holds the sum C_v. Then, the behavior search processing unit 14 calculates the difference between the sum C_{uv} and C_v, and sets the calculated difference value as the replacement cost.

[0067] The replacement cost is not limited to T / 2, but may be calculated by taking the sum or weighted sum of nodes at positions T / 2+1 and T / 2-1 as base points, for example.

[0068] After calculating the replacement cost of the intermediate node, the behavior search processing unit 14 compares the replacement cost with a preset threshold TH2 and judges whether the replacement cost is greater than the threshold TH2. If the result of this judgment is that the replacement cost is greater than the threshold TH2, the behavior search processing unit 14 adds the reference behavior feature information V used for comparison to the search behavior information.

[0069] Furthermore, prior to the above-mentioned search process for behavioral feature information, a centroid c_k of v is constructed by clustering v_{T / 2} for the reference behavioral feature information group, and when executing the hierarchical search process, the centroid c_k is compared with the cost C_{cu} when it is replaced with the intermediate node u_{T / 2} of the extracted behavioral feature information. If the cost compared with the centroid c_k is smaller than the threshold as a result of this comparison, the comparison process with other nodes is omitted and the search process ends.

[0070] [Other embodiments] In the first embodiment, the functions of the specific behavior detection device BD are provided in an information processing device such as a server computer or a personal computer that is provided independently of the camera CM and the terminal MT. However, the present invention is not limited to this, and all or part of the functions of the specific behavior detection device BD may be provided in the camera CM and the terminal MT.

[0071] In addition, in the first embodiment, the case of detecting a specific behavior of a person has been described as an example, but the detection target is not limited to a person, and may be an animal, a robot, etc. In addition, the type and configuration of the specific behavior detection device, the processing procedure and processing contents of each processing unit, etc. can be modified in various ways without departing from the gist of the present invention.

[0072] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiment may be appropriately adopted.

[0073] In short, the present invention is not limited to the above-mentioned embodiments as they are, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining the multiple components disclosed in the above-mentioned embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0074] BD: Specific behavior detection device CM…Camera MT…Terminal 1...Control unit 2...Program memory section 3. Data storage section 4... Input / Output I / F section 5. Bus 11...Video data acquisition processing unit 12...Human-related area image detection processing unit 13...Behavioral feature extraction processing unit 14...Behavior search processing unit 15…Search behavior ranking processing section 16...Behavioral evaluation processing unit 31...Video data storage unit 32...Behavioral characteristic information storage unit

Claims

1. A first processing unit that acquires image data of an image of a target area; a second processing unit that detects a partial region image including at least an object to be detected from the acquired frames of the image data; a third processing unit that extracts, for each first section including a plurality of the frames, feature amount information that indicates a feature of a movement of the object from the plurality of partial area images detected in the first section, and constructs a graph in which each of the extracted plurality of feature amount information is assigned to a number of nodes corresponding to a second section, thereby structuring the plurality of feature amount information, thereby generating first behavior feature information; a fourth processing unit that calculates a cost reflecting a degree of similarity between the generated first behavior feature information and each of a plurality of second behavior feature information related to the object that are prepared in advance, and detects the second behavior feature information for which the calculated cost satisfies a preset condition as search behavior information; a fifth processing unit that outputs the detected search behavior information; Equipped with The fourth processing unit is a process of mutually replacing a first intermediate node and a second intermediate node between the first behavior characteristic information and each of the plurality of second behavior characteristic information; A process of calculating first similarity information representing a sum of similarities between the replaced second or first intermediate node and each other node of the first or second behavioral feature information; A process of calculating second similarity information representing a sum of similarities between the second or first intermediate node of the second or first behavioral feature information and each of the other nodes of the second or first behavioral feature information; A process of calculating a difference between the first similarity information and the second similarity information as a substitution cost; a process of detecting the second behavior feature information, the substitution cost of which satisfies a preset condition, as the search behavior information; A specific behavior detection device comprising:

2. The specific behavior detection device according to claim 1 , wherein the second processing unit detects, as the partial area image, an image of an area including a moving part of the object to be detected and a surrounding object to be operated by the moving part.

3. 2. The specific behavior detection device according to claim 1, wherein the fourth processing unit compares the replacement cost with a predetermined threshold and detects, as the search behavior information, the second behavior characteristic information whose replacement cost is smaller than the threshold, or the second behavior characteristic information whose replacement cost is larger than the threshold.

4. The specific behavior detection device according to claim 1 , wherein the fifth processing unit is configured to rank the search behavior information based on the substitution cost.

5. A specific behavior detection method executed by an information processing device, acquiring image data of an area of ​​interest; detecting a partial area image including at least a detection target object from the acquired frames of image data; extracting, for each first section including a plurality of the frames, feature information representative of a feature of a movement of the object from the plurality of partial region images detected in the first section, and constructing a graph in which each of the extracted plurality of feature information is assigned to a number of nodes corresponding to a second section, thereby structuring the plurality of feature information, thereby generating first behavior feature information; calculating a cost reflecting a degree of similarity between the generated first behavior feature information and each of a plurality of second behavior feature information related to the object that are prepared in advance, and detecting the second behavior feature information whose calculated cost satisfies a preset condition as search behavior information; outputting the detected search behavior information; Equipped with The step of detecting the search behavior information includes: a step of mutually replacing a first intermediate node and a second intermediate node between the first behavior characteristic information and each of the plurality of second behavior characteristic information; calculating first similarity information representing a sum of similarities between the replaced second or first intermediate node and each other node of the first or second behavioral feature information; calculating second similarity information representing a sum of similarities between the second or first intermediate node of the second or first behavioral feature information and each of the other nodes of the second or first behavioral feature information; calculating a difference between the first similarity information and the second similarity information as a substitution cost; detecting, as the search behavior information, the second behavior feature information, the substitution cost of which satisfies a preset condition; A specific behavior detection method comprising:

6. 5. A program for causing a processor included in a specific behavior detection device to execute a process performed by at least one of the first processing unit to the fifth processing unit included in the specific behavior detection device according to claim 1.

Images