Apparatus, method and program for detecting video of interest

Abstract

According to one aspect of the present invention, object detection information including information representing an object and the position of the object in a video frame of video data captured by a camera is detected from the video frame, and distance detection information representing the distance distribution from the camera to the image drawn in the video frame is detected from the video frame. Then, a specific video region, in which a first object that is a target of interest and a second object that is a work target of the first object exist within a predetermined distance range, is detected on the basis of the object detection information and the distance detection information, and in the detected specific video region, a specific video region that satisfies a preset detection condition is detected as a video-of-interest.

Description

[Technical field]

[0001] One aspect of the present invention relates to an attention video detection device, method, and program for detecting an attention video including an object of interest, etc., from video data. [Background technology]

[0002] In recent years, with the spread of high-definition cameras, various technologies have been proposed to analyze human behavior based on captured images. For example, there are technologies that detect criminal behavior based on surveillance camera images and dangerous behavior at construction sites.

[0003] In order to detect human behavior with high accuracy, it is generally necessary to observe a large amount of video. In such cases, since manual observation is time-consuming and labor-intensive, research is being conducted on technologies that use algorithms to automatically detect specific behaviors. In addition, research is being actively conducted on the robustness of human tracking technology and face recognition technology to detect the movements of multiple people individually (for example, see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Z. Xiao et al. “Improving Transferability of Adversarial Patches on Face Recognition with Generative Models.” CVPR2021. Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, for example, to accurately detect the movement of a person, it is necessary to consider the interaction between the person and the surrounding objects that the person is working on, that is, the correlation between the movements. However, the technology described in Non-Patent Document 1 detects the movement of a specific person or object to be monitored by itself. For this reason, there are cases where the behavior of a person cannot be detected with high accuracy.

[0006] For example, scenes involving interactions between a person and objects, such as climbing a ladder or hammering a nail with a hammer, have unique potential dangers associated with surrounding objects, such as falling off a ladder or accidentally hitting a hand with the hammer. However, if only the movement of a person is detected as a monitoring target, it is not possible to monitor the potential dangers of the person's actions.

[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables accurate detection of an image of interest that is necessary for estimating the movement of an object that is the target of attention. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, one aspect of an attention video detection device or method according to the present invention includes a first processing unit or process for acquiring video data captured by a camera, a second processing unit or process for detecting object detection information including information representing an object and a position of the object in the video frame from a video frame of the video data, and a second processing unit or process for detecting object detection information including information representing a distance distribution from the camera of an image depicted in the video frame from the video frame of the video data. By extracting depth information and calculating the distance of the object from the camera based on the extracted depth information, A third processing unit or process for detecting distance detection information, and a first object to be noticed and a second object to be worked on by the first object based on the object detection information and the distance detection information. and a combination of the first object and the second object having a relationship in which a difference in distance from the camera between the first object and the second object is equal to or less than a predetermined value, and information representing the difference in distance. A fourth processing unit or step for detecting a specific image area, The combination of the first object and the second object included in the above and the difference in distance between them are and a fifth processing unit or step of detecting a specific image region that satisfies a preset detection condition as an image of interest.

[0009] According to one aspect of the present invention, for example, a first object designated as a target of attention and a second object surrounding the first object as a target of work are included, and a distance between the first object and the second object is determined. A specific image region including a combination of the first object and the second object having a relationship in which the difference in distance from the camera is equal to or less than a predetermined value and information representing the difference in distance As a result, when analyzing the movement of the first object based on the attention image, the distance between the first object and the surrounding second object that is the work target is detected. Depth direction Taking the interaction into account, it becomes possible to estimate the movement of the first object with high accuracy. Effect of the Invention

[0010] That is, according to one aspect of the present invention, it is possible to provide a technique that enables accurate detection of an image of interest that is necessary for estimating the movement of an object that is a target of attention. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an attention video detection device according to an embodiment of the present invention, together with the configuration of the peripheral parts. [Diagram 2] FIG. 2 is a block diagram showing an example of a software configuration of the interest video detection device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart showing an example of a processing procedure and processing contents of an attention video detection process executed by the control unit of the attention video detection device shown in FIG. [Figure 4] FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the object detection processing in the interest video detection processing shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing an example of the processing procedure and processing contents of the distance information detection processing in the interest video detection processing shown in FIG. [Figure 6A] FIG. 6A is a flowchart showing the processing procedure and processing contents of the first half of the specific section determination processing of the interest video detection processing shown in FIG. [Figure 6B]FIG. 6B is a flowchart showing the processing procedure and processing contents of the latter half of the specific section determination processing in the interest video detection processing shown in FIG. [Figure 7] FIG. 7 is a flowchart showing an example of a processing procedure and processing contents of the attention video detection processing shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of an attractive video detection rule stored in the attractive video detection related information storage unit shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [One embodiment] (Configuration example) FIG. 1 is a block diagram showing an example of the hardware configuration of an attention video detection device according to an embodiment of the present invention together with its peripheral devices, and FIG. 2 is a block diagram showing an example of the software configuration of the attention video detection device.

[0014] The interest video detection device BD is composed of an information processing device such as a server computer or a personal computer, and a camera CM and a terminal MT are connected to this interest video detection device BD via a signal cable or a network (not shown).

[0015] The camera CM is installed, for example, on a ceiling or wall surface capable of capturing an image of the area to be monitored, captures the entire body of a person to be monitored who is in the area to be monitored together with the surrounding area, and transmits the time-series image data to the image detection device BD.

[0016] The video data captured by the camera CM may be directly transmitted from the camera CM to the attention video detection device BD, or may be temporarily stored in a video database (not shown) and then transmitted to the attention video detection device BD. The number of cameras CM is not limited to one, and may be multiple.

[0017] The terminal MT is used, for example, by a system administrator or a monitor who monitors the behavior of the person being monitored, and is configured as an information processing terminal such as a personal computer. The terminal MT has a function of receiving, for example, attention video information detected by the attention video detection device BD, and analyzing the behavior of the person being monitored based on the received attention video information. Note that the function of analyzing the behavior of the person being monitored based on the attention video may be provided by the attention video detection device BD.

[0018] In addition, in the case where the interesting video detection device BD is equipped with a machine learning model, the terminal MT may have a function of inputting learning data required for training the machine learning model to the interesting video detection device BD.

[0019] Attention video detection device BD includes a control unit 1 using hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU), and is connected to a storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, the interface will be abbreviated as I / F) unit 4 via a bus 5. Note that control unit 1 may be configured using a programmable logic device (PLD), a field programmable gate array (FPGA), or the like.

[0020] The input / output I / F unit 4 has a communication interface function, and transmits and receives video data and various input / output data between the camera CM and the terminal MT via a signal cable or a network.

[0021] The program storage unit 2 is configured by combining a non-volatile memory such as a solid state drive (SSD) that can be written and read at any time as a storage medium, and a non-volatile memory such as a read only memory (ROM), and stores application programs required for executing various control processes according to an embodiment, in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

[0022] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and is equipped with a video data storage unit 31, a specific section video determination related information storage unit 32, and a noteworthy video detection related information storage unit 33 as main storage units required to implement one embodiment.

[0023] The video data storage unit 31 is used to temporarily store the time-series video data transmitted from the camera CM for subsequent interest video detection processing.

[0024] The specific section video assessment related information storage unit 32 stores specific section video assessment related information including the names of a designated area and a designated object that are designated in advance as monitoring targets.

[0025] For example, when a surveillance camera CM is fixedly installed in a surveillance area, the designated area is automatically set by the terminal MT or the interesting video detection device BD using a method such as semantic segmentation based on the video of the camera CM. The designated area may also be set manually by an administrator or a surveillance person from the terminal MT.

[0026] The designated area is expressed by a shape such as a rectangle, a polygon, or a circle. The designated area is not limited to one, and multiple areas can be designated. For example, if the shape is a rectangle, multiple rectangles are designated as detection target areas within a video frame captured by one fixed camera CM. It is not necessary to set a designated area. In this case, the entire video frame is considered to be the designated area.

[0027] On the other hand, the designated object is the name of an object to be detected, such as a person, a ladder, a hammer, a safety belt, etc. Note that the designated object does not necessarily have to be designated. In this case, all objects are regarded as detection targets.

[0028] Attention video detection related information storage unit 33 stores detection rules necessary for detecting attention videos. The detection rules define rules for determining whether or not a video of a specific section detected by specific section determination processing unit 14 described later is attention video in which an interaction between a person and an object is likely to occur, and multiple detection rules are stored in advance.

[0029] The control unit 1 includes, as processing functions necessary for implementing one embodiment, a video data acquisition processing unit 11, an object detection processing unit 12, a distance information detection processing unit 13, a specific section determination processing unit 14, and a target video detection processing unit 15. All of the above processing units 11 to 15 are realized by causing a hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

[0030] The video data acquisition processing unit 11 acquires time-series video data output from the camera CM via the input / output I / F unit 4, and performs processing to temporarily store each of the acquired video data in the video data storage unit 31.

[0031] The object detection processing unit 12 includes a machine learning model for object detection constructed using a neural network. The object detection processing unit 12 reads video data from the video data storage unit 31 frame by frame, and inputs the read video frames to the machine learning model for object detection, thereby detecting the class and position of an object appearing in the video frame as object detection information. An example of this object detection process will be described in the operation example.

[0032] The distance information detection processing unit 13 includes a machine learning model for detecting distance information that is constructed using a neural network. The distance information detection processing unit 13 reads video data from the video data storage unit 31 frame by frame, and inputs the read video frames to the machine learning model for detecting distance information, thereby detecting the distance from the camera CM of the image shown in the video frame in pixel units. An example of the distance information detection process will also be described in the operation example.

[0033] The specific section determination processing unit 14 receives object detection information and distance detection information from the object detection processing unit 12 and the distance information detection processing unit 13, respectively. Then, by comparing these detection information with the specific section video determination related information stored in the specific section video determination related information storage unit 32, a process is performed to detect a video of a specific section in which a specified object appears within a specified area. An example of the specific section detection process will be described in more detail in the operation example.

[0034] Attention video detection processing unit 15 receives the specific section video information detected by the specific section determination processing unit 14. Then, by comparing the received specific section video information with detection rules stored in attention video detection related information storage unit 33, it determines whether the specific section video information is attention video or not, and if it is attention video, performs processing to output the attention video from input / output I / F unit 4 to terminal MT. An example of attention video determination processing will also be described in more detail in the operation example.

[0035] The machine learning model for object detection and the machine learning model for distance information detection are, for example, configured by a convolutional neural network, but the type of neural network can be appropriately selected and used. Also, in the specific section determination processing unit 14 and the attention video detection processing unit 15, the specific section determination process and the attention video detection process may be performed using a machine learning model, respectively.

[0036] (Example of operation) Next, an example of the operation of the interest video detection device BD configured as above will be described. In the following description, it is assumed that the machine learning model for object detection and the machine learning model for distance information detection have been trained in advance.

[0037] FIG. 3 is a flowchart showing an example of the overall processing procedure and processing contents of the attention video detection processing executed by the control unit 1 of the attention video detection device BD.

[0038] (1) Acquisition of video data The control unit 1 of the target video detection device BD, under the control of the video data acquisition processing unit 11, acquires time-series video data obtained by photographing the area to be monitored by the camera CM via the input / output I / F unit 4. Then, the acquired video data is stored in the video data storage unit 31 sequentially for each frame.

[0039] (2) Object detection When the above-mentioned video data is acquired for a certain period of time, the control unit 1 of the interest video detection device BD, under the control of the object detection processing unit 12, performs a process in step S1 to detect information representing the object class and position for each frame from the video data as follows.

[0040] FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the object detection processing executed by the object detection processing unit 12.

[0041] That is, in step S11, the object detection processing unit 12 reads the video data for each frame from the video data storage unit 31. Then, the read video frames are input to a machine learning model for object detection, and information representing the class and position of each object captured in the video frame is detected by the machine learning model.

[0042] For example, if "person", "ladder", and "safety belt" have been specified in advance as objects to be detected, then in step S12, the object detection learning model of the object detection processing unit 12 performs a detection process for each of the above objects from the video frame, and outputs class information represented by a binary vector in which a detected object is "1" and an undetected object is "0". For example, if a "person" and a "ladder" are detected in a current video frame, the class information (1,1,0) is output. Note that the vector length depends on the number of object classes.

[0043] On the other hand, the position of an object is represented by bounding box information. The bounding box information is represented by (x_min, y_min, x_max, y_max). This indicates the coordinates of the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max) of a rectangle when the coordinates of the upper left corner of the video frame are (0, 0). The learning model for object detection of the object detection processing unit 12 outputs bounding box information for each object included in the class information in step S13.

[0044] The object detection processing unit 12 outputs object detection information consisting of the class information and bounding box information output from the learning model for object detection to the specific section determination processing unit 14 in step S14.

[0045] In order to maintain the chronological consistency of a person, a person ID for identifying a person may be generated using a known person tracking technique, and the person ID may be included in the object detection information. The person tracking technique is described in detail in, for example, K. Zhou et al. "Omni-Scale Feature Learning for Person Re-Identification." ICCV2019. Also, a known technique may be used to detect an object from a video frame.

[0046] (3) Extraction of distance information In parallel with the above object detection process, the control unit 1 of the interest video detection device BD, under the control of the distance information detection processing unit 13, executes a process in step S2 to extract the distance from the camera CM of the image captured in the above video frame in pixel units as follows.

[0047] FIG. 5 is a flowchart showing an example of the procedure and content of distance information detection processing executed by distance information detection processing unit 13.

[0048] That is, in step S21, the distance information detection processing unit 13 first reads the video data for each frame from the video data storage unit 31. Then, the read video frames are input to a machine learning model for distance information detection, and information representing a distance distribution for each pixel from the camera CM of the image captured in the video frames is detected and output by this learning model.

[0049] For example, the distance distribution is represented by a depth frame D. Depth frame D contains, for each pixel p(u, v) in video frame I, corresponding depth information d(u, v), where p and d are functions that return the pixel value and depth value corresponding to each pixel position on the video frame, and u and v represent the pixel position on the video frame.

[0050] In step S22, the machine learning model for distance information detection extracts a depth frame D corresponding to the input video data, and outputs the extracted depth frame D to the specific section determination processing unit 14 as distance detection information in step S23.

[0051] If a depth camera or the like is used as the camera CM, the depth frame may be acquired from the depth camera. However, in this case, the depth frame acquired from the depth camera needs to be aligned with the video frame in time and space.

[0052] (4) Determination of specific sections Next, in step S3, the control unit 1 of the interest video detection device BD, under the control of the specific section determination processing unit 14, executes a process of determining a specific section in which the specified object is captured in a specified area in the video frame based on the above object detection information and distance detection information, as follows.

[0053] 6A and 6B are flowcharts showing an example of the processing procedure and processing contents of the specific section determination processing executed by the specific section determination processing unit 14. In FIG.

[0054] (4-1) Detection of specified areas and objects 6A, the specific section determination processing unit 14 first receives the object detection information and the distance detection information from the object detection processing unit 12 and the distance information detection processing unit 13, respectively. Then, in step S312, the specific section video determination related information is read from the specific section video determination related information storage unit 32.

[0055] Next, in step S313, the specific section determination processing unit 14 compares the object detection information with the specific section video determination related information and determines whether any object is captured in the designated area of ​​the detection target defined in the specific section video determination related information. If any object is captured in the designated area, the specific section determination processing unit 14 extracts the entire designated area as object area information.

[0056] Next, in step S314, the specific section determination processing unit 14 determines whether or not there is object region information in which a person is photographed among the plurality of detected object region information, and ends the process if there is no corresponding region information. On the other hand, if object region information in which a person is photographed is found, the specific section determination processing unit 14 next determines in step S315 whether or not there is object region information in which an object other than a person is photographed among the object region information in which the person is photographed, and ends the process if there is no corresponding object region information.

[0057] On the other hand, if object region information containing an object is detected, then in step S316, specific section determination processing unit 14 extracts designated object region information containing an object that matches the object specified as the detection target by the specific section video determination related information.

[0058] By performing the series of processes in steps S311 to S316 described above, designated object area information is detected in which the designated area contains at least one person area containing a person, and one or more object areas containing a designated object that may be a work target for the person.

[0059] (4-2) Detection of specific video segments based on distance Next, in step S321 of FIG. 6B, the specific section determination processing unit 14 extracts distance detection information corresponding to the specified object region information from the distance detection information previously received in step S311. That is, distance detection information corresponding to each of a person region in which a person is photographed and an object region in which a specified object is photographed within the specified region is extracted. Here, the distance detection information corresponding to the person region and the object region is included as depth information in a depth frame corresponding to the bounding box of each region. That is, the distance detection information corresponding to each of the person region and the object region is a set of depth information d(u, v) included in a depth frame corresponding to a bounding box (x_min, y_min, x_max, y_max).

[0060] The specific section determination processing unit 14 then calculates the average distance for each of the person region and the object region in step S322. This average distance is calculated as the average of a set of depth information d(u, v) included in the depth frame corresponding to the bounding box (x_min, y_min, x_max, y_max).

[0061] Next, in step S323, the specific section determination processing unit 14 generates a list H of N person regions in the specified object region information and a list O of M object regions other than person regions. Then, in step S324, the specific section determination processing unit 14 initializes i of the person region h_i included in the list H to i=0, and in step S325, initializes j of the object region o_j included in the list O to j=0, and then in step S326, calculates a difference E between the average distance of the person region h_0 included in the list H and the average distance of the object region o_0 included in the list O.

[0062] Next, in step S327, specific section determination processing unit 14 compares the calculated average distance difference E with a threshold T to determine whether or not the average distance difference E is equal to or less than threshold T. If the result of this determination is that the average distance difference E is equal to or less than threshold T, in step S328, the pair of person region h_0 and object region o_0 and the average distance difference E are added to the specific section video information. Note that if the average distance difference E exceeds threshold T, the above addition is not performed and the process proceeds to step S329.

[0063] Next, in step S329, specific section determination processing unit 14 increments j of the object region information o_j by j+1 to select the next object region information o_1, returns to step S326, and calculates the difference E between the average distance of the object region o_1 and the average distance of the person region h_0. Then, in step S327, it is determined whether or not the calculated difference E is equal to or smaller than a threshold value T, and if the difference E is equal to or smaller than the threshold value T, the pair of the person region information h_0 and object region information o_1 and the average distance difference E are added to the specific section video information, but if the difference E exceeds the threshold value T, they are not added.

[0064] Thereafter, in the same manner, in steps S326 to S330, specific section determination processing unit 14 repeatedly executes the process of sequentially calculating the average distance difference E between all M object regions o_j (j=2, ..., M) included in the object region information list O and the person region h_0, and comparing it with threshold value T. Then, if a pair of object region o_j (j=2, ..., M) and person region h_0 for which the difference E is equal to or smaller than threshold value T is found, this pair is added to the specific section video information together with the average distance difference E.

[0065] Next, in step S331, specific section determination processing unit 14 increments the value of i of person area h_i (i+1) to select the next person area h_1. Then, returning to step S325 to initialize j of object area o_j, in step S326, calculates difference E between the average distance of person area h_1 and the average distance of object area o_0, and in step S327, determines whether or not difference E is equal to or less than threshold value T. Then, if difference E is equal to or less than threshold value T, the pair of person area h_1 and object area o_0 and the difference E of the average distance are added to specific section video information.

[0066] Thereafter, the specific section determination processing unit 14 similarly repeats the process of calculating the average distance difference E for each combination of the person region h_1 and all M object regions o_j (j=1, ..., M) and comparing it with the threshold value T. Then, when a combination of the person region h_1 and the object region o_j (j=1, ..., M) for which the average distance difference E is equal to or smaller than the threshold value T is found, the specific section determination processing unit 14 adds this combination and the average distance difference E to the specific section video information.

[0067] That is, in steps S325 to S332, specific section determination processing unit 14 repeatedly executes the process of calculating the average distance difference E for all combinations of all N person regions h_i (i=0, ..., N) and all M object regions o_j (j=0, ..., M) and comparing the calculated average distance difference E with threshold value T. Then, when a combination of person region h_i (i=0, ..., N) and object region o_j (j=0, ..., M) for which the average distance difference E is equal to or less than threshold value T is found, this combination is added to the specific section video information together with the average distance difference E.

[0068] In the above series of processes, the difference E in the average distance between the person area h_i (i=0,...,N) and the object area o_j (j=0,...,M) is used to determine whether or not to add it to the specific section video information, because the smaller the difference in the average distance, the more likely it is that an interaction is occurring between a person and an object.

[0069] Finally, in step S333, specific section determination processing section 14 outputs the specific section video information obtained by the above series of processes to attention video detection processing section 15.

[0070] (5) Detection of interesting images Next, in step S4, the control section 1 of the attractive video detection device BD, under the control of the attractive video detection processing section 15, executes a process of detecting an attractive video from the specific section video information as follows.

[0071] FIG. 7 is a flowchart showing an example of the processing procedure and processing content of the attention video detection processing executed by attention video detection processing unit 15.

[0072] That is, first, in step S41, the attention video detection processing unit 15 receives specific section video information from the attention video detection processing unit 14. Next, in step S42, the attention video detection processing unit 15 reads attention video detection related information from the attention video detection related information storage unit 33. Then, in step S43, the attention video detection processing unit 15 checks the specific section video information against the detection rule defined by the attention video detection related information, and determines whether or not there is specific section video information that matches the detection rule.

[0073] For example, assume that detection rules as shown in Fig. 8 are defined in the attention video detection related information. Attention video detection processing unit 15 judges whether or not the combination of the person region and the object region and the difference in the average distance for each specific section video included in the specific section video information correspond to any of the above detection rules. This judgment also takes into consideration the distance on the video frame between the person region and the object region. Note that the distance on the video frame between the person region and the object region is expressed as the Euclidean distance between the center point of the bounding box of the person region and the center point of the bounding box of the object region.

[0074] If a specific section video that meets any of the detection rules is found as a result of the above determination, attention video detection processing unit 15 outputs the specific section video as an attention video from input / output I / F unit 4 to terminal MT in step S44. On the other hand, specific section videos that do not meet any of the above detection rules are discarded.

[0075] (Action and effect) As described above, in one embodiment, for each frame of video data acquired from a camera CM, the class and position of an object captured in the video frame are detected as object detection information, and the distance distribution of the image captured in the video frame from the camera CM is detected as distance detection information. Then, from the object detection information, all object area information including a person and an object to be worked on, which is included in a designated area defined as a detection target by the specific section video related information, is detected, and further, based on the distance detection information, all specific section videos including a pair of a person and an object in which the difference in the average distance between the person and the object is equal to or less than a threshold are detected from each of the object area information. Then, from each detected specific section video, a specific section video that corresponds to a detection rule previously defined by the attention video detection related information is detected, and the detected specific section video is output as an attention video.

[0076] Therefore, according to one embodiment, a video including a person designated as a detection target and a surrounding object that the person is working on, and in which the distance between the person and the object is within a threshold, is detected as a video of interest. As a result, when analyzing the behavior of a person based on the video of interest, it is possible to estimate the behavior of the person with high accuracy by taking into account the interaction between the person and the surrounding object that is the target of the person's work.

[0077] [Other embodiments] In the above embodiment, an example was described in which an image area including a pair of a person and a surrounding object that is a work target, in which the difference in distance between the person and the surrounding object is equal to or less than a threshold, is set as the image of interest. However, the detection target is not limited to a combination of a person and an object, and it is also possible to detect, for example, an image including a combination of a person to be monitored and another person who works with the person, or a combination of an animal or a robot and a surrounding object related to the animal or robot, as the image of interest.

[0078] In addition, in the above embodiment, an example was given of a case where an image in which the difference in distance between a person and surrounding objects is below a threshold value is set as the image of interest, but an image that satisfies the above condition may be set as the image of interest when the image continues for a predetermined period of time or longer.

[0079] In the embodiment, the functions of the attention video detection device BD are provided in an information processing device such as a server computer or a personal computer that is provided independently of the camera CM and the terminal MT. However, the present invention is not limited to this, and all or part of the functions of the attention video detection device BD may be provided in the camera CM and the terminal MT. Furthermore, the type, function, processing procedure, processing contents, etc. of the interest image detection device BD can be modified in various ways without departing from the spirit and scope of the present invention.

[0080] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiment may be appropriately adopted.

[0081] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0082] BD…Attention video detection device CM…Camera MT…Terminal 1...Control unit 2...Program memory section 3. Data storage section 4... Input / Output I / F section 5. Bus 11...Video data acquisition processing unit 12...Object detection processing unit 13...Distance information detection processing unit 14...Specific section determination processing unit 15...Attention image detection processing section 31...Video data storage unit 32...specific section video judgment related information storage unit 33... Attention image detection related information storage unit

Claims

1. A first processing unit that acquires video data captured by a camera; a second processing unit that detects object detection information from a video frame of the video data, the object detection information including information representing an object and a position of the object in the video frame; a third processing unit that extracts depth information representing a distance distribution from the camera of an image of the object depicted in the video frame from the video frame of the video data, and obtains distance detection information by calculating a distance of the object from the camera based on the extracted depth information; a fourth processing unit that detects a specific image area including a first object that is an attention target and a second object that is a work target of the first object based on the object detection information and the distance detection information, the specific image area including a combination of the first object and the second object, the difference in distance between the first object and the second object from the camera being equal to or smaller than a predetermined value, and information representing the difference in distance; a fifth processing unit that detects, as a target image, a combination of the first object and the second object included in the specific image area and the specific image area in which a difference in distance between the combination and the second object satisfies a preset detection condition; An attention video detection device comprising:

2. The video interest detection device according to claim 1 , wherein the second processing unit outputs, as the object detection information, class information representing the presence or absence of the object and bounding box information representing an area in the video frame in which the object exists.

3. 3. The video attention detection device according to claim 1, wherein the first object is a person, and the second object is an object that ...

4. An attention video detection method performed by an information processing device, acquiring image data captured by a camera; detecting object detection information from a video frame of the video data, the object detection information including information representing an object and a position of the object in the video frame; extracting depth information representing a distance distribution from the camera of an image of the object depicted in the video frame from the video frame of the video data, and calculating a distance of the object from the camera based on the extracted depth information to obtain distance detection information; a step of detecting a specific image area including a first object to be focused on and a second object to be worked on by the first object based on the object detection information and the distance detection information, the specific image area including a combination of the first object and the second object, the difference in distance between the first object and the second object from the camera being equal to or smaller than a predetermined value, and information representing the difference in distance; detecting, as a target image, a specific image region in which a combination of the first object and the second object included in the specific image region and a difference in distance therebetween satisfy a preset detection condition; An interest video detection method comprising:

5. A program that causes a processor provided in a video detection device of interest described in any one of claims 1 to 3 to execute processing performed by at least one of the first processing unit to the fifth processing unit provided in the video detection device of interest.

Images