Motion recognition system, motion recognition method, and motion recognition program

Abstract

To perform high-accuracy operation recognition using a small number of query images.SOLUTION: An input reception unit 2 reproduces and displays a first moving image obtained by imaging a person who performs specific action, and receives instruction input to instruct a position related to the specific action on frames 10a and 10b of the reproduced and displayed first moving image during reproduction display. A determination unit 3 calculates similarity between the first moving image and a second moving image on the basis of a comparison result of a feature amount between a second feature point group included in the second moving image and a first feature point group included in the first moving image to determine whether or not the specific action is included in the second moving image on the basis of the calculated similarity. The determination unit 3 provides larger weight for the comparison result of the feature amount at the feature points closer to the instructed position by the instruction input among the first feature point group when calculating the similarity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a motion recognition system, a motion recognition method, and a motion recognition program. [Background technology]

[0002] The technology to automatically recognize specific human actions from images is expected to be applied to various fields. For example, it is conceivable to install cameras in a store that sells goods and automatically determine customer actions from the captured images. In this case, applications such as realizing unmanned stores by detecting customers' purchase actions and automatically settling them, detecting fraudulent or dangerous behavior of customers, and optimizing product displays and prices by analyzing customer behavior are possible.

[0003] Regarding automatic recognition of actions, for example, the following image search device has been proposed: This image search device recognizes posture information of the search target from an input image, extracts features from the posture information and the input image, associates them with the input image, stores them in a database, generates a search query from posture information specified by the user, and searches the database for images containing similar postures according to the search query.

[0004] There are also techniques for automatically recognizing actions by learning from images. For example, a method has been proposed in which a system learns from a large number of training images in which the positions and types of actions to be detected in the images are specified, and then uses the learning results to estimate the human actions and their types in unknown test images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-91138 A [Non-patent literature]

[0006] [Non-Patent Document 1] Chunhui Gu, 11 others, AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions, IEEE Conference on Computer Vision and Pattern Recognition(CVPR) 2018 Summary of the Invention [Problem to be solved by the invention]

[0007] Incidentally, when determining whether a specific action is captured in a query image and an image to be recognized based on the similarity between the latter image, for example, feature amounts are compared between feature points of each image. Here, the structure of the human body is complex, and even when a person performs a certain action, the whole body may take various postures. Therefore, there is a problem that the recognition accuracy of a certain action cannot be improved unless a large number of query images in which the specific action is performed in various postures are used.

[0008] In one aspect, the present invention aims to provide an action recognition system, an action recognition method, and an action recognition program capable of performing highly accurate action recognition using a small number of query images. [Means for solving the problem]

[0009] In one proposal, a motion recognition system is provided that includes an input receiving unit and a determination unit. In this motion recognition system, the input receiving unit plays and displays a first video of a person performing a specific motion, and receives, during playback and display, an instruction input that indicates a position related to the specific motion on a frame of the first video. The determination unit calculates a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determines whether or not the second video includes the specific motion based on the calculated similarity. When calculating the similarity, the determination unit assigns a larger weight to the comparison result of feature amounts for feature points in the first feature point group that are closer to a position indicated by the instruction input.

[0010] Also, one proposal provides a method for recognizing an action, in which a computer executes a process of playing back and displaying a first video of a person performing a specific action, accepting, during playback and display, an instruction input indicating a position related to the specific action on a frame of the first video, calculating a similarity between the first video and the second video based on a comparison result of features between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determining whether or not the second video includes the specific action based on the calculated similarity. In this method for recognizing an action, in the calculation of the similarity, a larger weight is assigned to the comparison result of features of the first feature point group that is closer to the position indicated by the instruction input.

[0011] Furthermore, in one proposal, a motion recognition program is provided that causes a computer to execute a process similar to the motion recognition method described above. Effect of the Invention

[0012] In one aspect, highly accurate action recognition can be performed using a small number of query images. [Brief description of the drawings]

[0013] [Figure 1] 1 illustrates a configuration example and a processing example of an action recognition system according to a first embodiment; [Diagram 2] FIG. 11 illustrates an example of a configuration of an information processing system according to a second embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 4] 2 is a block diagram showing an example of the configuration of processing functions included in an information processing device; [Diagram 5] 1 is an example of a flowchart showing an overall processing procedure of a motion detection and analysis process performed by an information processing device. [Figure 6] FIG. 11 is a diagram for explaining a motion feature amount; [Figure 7] 13 is an example of a flowchart showing a process of calculating a motion feature amount from a sample video. [Figure 8] FIG. 11 is a diagram showing a first example of motion features. [Figure 9] FIG. 13 is a diagram showing a third example of motion features. [Figure 10] FIG. 13 is a diagram showing an example of a time-space instruction input. [Figure 11] FIG. 11 is a diagram for explaining a calculation process of an attention level. [Figure 12] 13 is an example of a flowchart illustrating a calculation process of an attention level according to a time-space instruction input. [Figure 13] 13 is an example of a flowchart showing a process of calculating a motion feature amount from a detection target moving image. [Figure 14] 11A and 11B are diagrams for explaining a similarity search process between a sample video and a detection target video. [Figure 15] 11 is a diagram for explaining a similarity calculation between an input section video image and a partial video image; FIG. [Figure 16] 13 is an example of a flowchart illustrating a motion detection process based on a similarity degree. [Figure 17] FIG. 13 is a diagram illustrating an example of an analysis table. [Figure 18] 13 illustrates an example of a configuration of processing functions included in an information processing device according to a third embodiment. [Figure 19] 13 is a flowchart illustrating an example of an overall processing procedure of a motion detection and analysis process according to the third embodiment. [Figure 20] 13 is a flowchart illustrating an example of a learning and re-determination process. [Figure 21] FIG. 13 illustrates an example of a configuration of an information processing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First Embodiment FIG. 1 is a diagram showing an example of the configuration and processing of a motion recognition system according to a first embodiment. The motion recognition system shown in FIG. 1 is a system that executes motion recognition processing to determine whether a person performing a specific motion is captured in a video image, and is realized, for example, as a computer. The motion recognition system 1 has an input receiving unit 2 and a determination unit 3. The processing of the input receiving unit 2 and the determination unit 3 is realized, for example, by a processor executing a predetermined program. Note that the input receiving unit 2 and the determination unit 3 may each be implemented in a separate device.

[0015] The input receiving unit 2 plays and displays a first video of a person performing a specific action. This first video is used as a query image in the action recognition process. The input receiving unit 2 receives an instruction input that indicates a position related to the specific action on a frame of the first video that is played and displayed.

[0016] In Fig. 1, the first moving image includes frames 10a, 10b, ... from the beginning. In frame 10a, a pointing input is made by, for example, a mouse cursor to point to a pointing position 11a, and in frame 10b, a pointing input is made to point to a pointing position 11b. In this way, the pointing input may be made continuously across a number of frames in the first moving image. Also, while the frames are being displayed one after the other, it is possible to change the pointing position, in which case the pointing position made by the pointing input is recorded for each frame.

[0017] As described above, the instruction input is made so as to indicate a position related to a specific motion of the recognition target. For example, the instruction input is made so as to indicate the vicinity of a body part of the person appearing in the first video that is highly related to the specific motion. In FIG. 1, it is assumed that the specific motion detected is a motion of a person picking up a product (clothing). In such a motion, the movement near the tip of the hand is distinctive in distinguishing it from other motions. Therefore, while viewing the first video being played back and displayed, the operator makes an instruction input so as to indicate the vicinity of the tip of the hand of the person appearing on the screen.

[0018] The determination unit 3 compares the feature amounts between the feature points included in the second video, which is the detection target for the specific action, and the feature points included in the first video, and calculates the similarity between the first and second videos based on the comparison result. The determination unit 3 determines whether the specific action is included in the second video based on the calculated similarity. In FIG. 1, the second video includes frames 20a, 20b, ... from the beginning.

[0019] For example, assume that body parts of a person are extracted as feature points. In the example of FIG. 1, assume that the positions of the palm and the ankle are extracted as feature points from each frame of the first video. For example, a feature point 12a corresponding to the palm and a feature point 12b corresponding to the ankle are extracted from frame 10a of the first video. In this case, the positions of the palm and the ankle are also extracted as feature points from each frame of the second video. For example, a feature point 22a corresponding to the palm and a feature point 22b corresponding to the ankle are extracted from frame 20a of the second video.

[0020] In calculating the similarity between the frames 10a and 20a, the features are compared between the feature points 12a and 22a, and between the feature points 12b and 22b. The similarity between the frames 10a and 20a is calculated based on the comparison results of the respective features.

[0021] Finally, the similarity between the first video and the second video is calculated for all combinations of frames between the first video and the second video, and the similarities are summed to obtain the similarity between the first video and the second video. The determination unit 3, for example, compares the similarity thus obtained with a predetermined threshold value to determine whether or not the second video includes a specific action.

[0022] However, when calculating the similarity, the determination unit 3 assigns a larger weight to the comparison result of the feature amounts of feature points included in the first moving image, the closer the feature points are to the pointing position by the pointing input. As a result, the feature amounts of feature points closer to the pointing position have a higher contribution to the similarity calculation.

[0023] 1, for example, since feature point 12a on frame 10a is close to pointing position 11a, a large weight is assigned to the comparison result of the feature amounts between feature point 12a and feature point 22a. On the other hand, since feature point 12b on frame 10a is far from pointing position 11a, a small weight is assigned to the comparison result of the feature amounts between feature point 12b and feature point 22b.

[0024] The structure of the human body is complex, and when the body is photographed, the appearance of the body in the photographed image changes in various ways. Even when a person performs a specific action to be detected, there is diversity in the appearance of the body as a whole. For example, when a person performs the action of picking up a product, the movements of the body parts near the hands will be similar, whereas the movements of the feet may differ greatly depending on the person and the situation. For this reason, when it is desired to detect the action of picking up a product from an image, information on feature points near the hands is highly useful for action detection, but information on feature points near the feet is less useful for action detection.

[0025] By the above-described process of the determination unit 3, the feature amount of a feature point closer to the pointing position by the instruction input has a higher contribution to the similarity calculation. Therefore, by simply preparing at least one first video image as a query image, even if there is diversity in the body movement when a specific movement is performed, the specific movement can be detected with high accuracy. That is, according to the first embodiment, it is possible to perform highly accurate movement recognition using a small number of query images.

[0026] Note that inputting instructions to the first video sequence can be realized by a simple operation using a mouse, for example. For example, by dragging the mouse, inputting instructions can be continuously performed across multiple frames included in the first video sequence.

[0027] Second Embodiment Next, as a second embodiment, a system including an information processing device having the action recognition processing function of the action recognition system 1 in FIG. 1 will be described.

[0028] Fig. 2 is a diagram showing an example of the configuration of an information processing system according to the second embodiment. The information processing system shown in Fig. 2 includes an information processing device 100 and a camera 200 connected to the information processing device 100.

[0029] The information processing device 100 is realized as a computer such as a personal computer or a server device. The information processing device 100 detects that a person captured in a moving image has performed a specific action based on the moving image captured by the camera 200. The camera 200 is placed at a predetermined position where a person may be captured, and transmits data of the captured moving image to the information processing device 100.

[0030] As shown in FIG. 2, in this embodiment, as an example, a camera 200 is placed in a store 210 where a product 211 is sold, and captures the behavior of a person 212 (customer) standing in front of the product 211. The information processing device 100 detects, based on the captured video, that the person 212 performs a specific behavior related to the product 211. For example, the information processing device 100 detects the behavior of the person 212 picking up the product 211 or the behavior of the person 212 checking the price of the product 211. Furthermore, the information processing device 100 can perform various analyses, such as customer behavior analysis, using the detection results of the behavior of the person 212 over a predetermined period of time.

[0031] Fig. 3 is a diagram showing an example of a hardware configuration of an information processing device. The information processing device 100 is realized, for example, as a computer as shown in Fig. 3. The information processing device 100 shown in Fig. 3 has a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a graphic interface (I / F) 104, an input interface (I / F) 105, a reading device 106, a communication interface (I / F) 107, and a network interface (I / F) 108.

[0032] The processor 101 performs overall control of the entire information processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0033] The RAM 102 is used as a main storage device of the information processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0034] The HDD 103 is used as an auxiliary storage device of the information processing device 100. An OS program, an application program, and various data are stored in the HDD 103. Note that other types of non-volatile storage devices such as a solid state drive (SSD) can also be used as the auxiliary storage device.

[0035] A display device 104a is connected to the graphic interface 104. The graphic interface 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. The display device may be a liquid crystal display or an organic EL (ElectroLuminescence) display.

[0036] An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. The input device 105a may be a keyboard or a pointing device. The pointing device may be a mouse, a touch panel, a tablet, a touch pad, a trackball, or the like.

[0037] A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits it to the processor 101. The portable recording medium 106a may be an optical disk, a semiconductor memory, or the like.

[0038] The communication interface 107 transmits and receives data to and from the camera 200. The network interface 108 transmits and receives data to and from other devices via a network 108a.

[0039] The processing functions of the information processing device 100 can be realized by the above hardware configuration. Meanwhile, the information processing device 100 detects a specific motion from a video in the following procedure. Prior to motion detection, the information processing device 100 acquires a motion (sample motion) from the camera 200 that captures a person actually performing the specific motion to be detected, and detects motion features indicating the characteristics of the motion from this sample motion. After that, the information processing device 100 acquires a motion that is the target of motion detection (detection target motion) and detects motion features from this detection target motion. Then, the information processing device 100 detects a section of the motion in which a person performing a specific motion is captured from the detection target motion by comparing the motion features between the sample motion and the detection target motion.

[0040] Fig. 4 is a block diagram showing an example of the configuration of processing functions of an information processing device. As shown in Fig. 4, the information processing device 100 includes a storage unit 110, a video input unit 111, a feature amount calculation unit 112, a time-space batch input unit 113, an attention calculation unit 114, a video search unit 115, a similarity calculation unit 116, and an analysis unit 117.

[0041] The storage unit 110 is realized by a storage area of ​​a storage device included in the information processing device 100, such as the RAM 102 and the HDD 103. The storage unit 110 stores sample video data 121 and detection target video data 131. The storage unit 110 also stores motion feature data 122 and attention level data 123 as data corresponding to the sample video data 121. The storage unit 110 also stores motion feature data 132 as data corresponding to the detection target video data 131.

[0042] The processing of the video input unit 111, feature calculation unit 112, time-space batch input unit 113, attention calculation unit 114, video search unit 115, similarity calculation unit 116 and analysis unit 117 is realized, for example, by the processor 101 executing a predetermined application program.

[0043] Data of a moving image captured by camera 200 is input to video input unit 111. As this moving image, the above-mentioned sample moving image and a detection target moving image are input. Feature amount calculation unit 112 calculates a motion feature amount from the moving image input to video input unit 111. The feature amount is calculated as data for each feature point extracted from each frame of the moving image.

[0044] When a sample video is input to video input unit 111, feature calculation unit 112 calculates motion features from the sample video, and stores this in storage unit 110 as motion feature data 122. At the same time, feature calculation unit 112 stores the data of the sample video as sample video data 121 in storage unit 110. When a detection target video is input to video input unit 111, feature calculation unit 112 calculates motion features from the detection target video, and stores this in storage unit 110 as motion feature data 132. At the same time, feature calculation unit 112 stores the data of the detection target video in storage unit 110 as detection target video data 131.

[0045] The time-space collective input unit 113 plays and displays the sample video input to the video input unit 111 on the display device. The time-space collective input unit 113 accepts instruction input from an operator while the sample video is being played and displayed. This instruction input is made to indicate the position of the body part of the person making the characteristic movement in the movement when the person in the sample video is performing the detection target movement. The instruction input continues while the movement is being performed, and ends when the movement ends. This allows the range of the section in the sample video where the specific movement is being performed (i.e., temporal information) and the position of the body part of the person making the characteristic movement (i.e., spatial information) to be specified collectively. Note that the instruction input is made by, for example, dragging the mouse.

[0046] Hereinafter, the instruction input of the operator accepted by the time-space batch input unit 113 will be referred to as a “time-space instruction input,” and the information specified by the instruction input will be referred to as “time-space instruction information.” The time-space instruction information includes the start time and end time of the time-space instruction input in the sample video, and the instruction position in each frame from the start time to the end time.

[0047] The attention level calculation unit 114 calculates the attention level for each feature point based on the designated position for each frame of the sample video during the section in which the time-space instruction input is being performed among the frames of the sample video. The attention level is used to weight the motion feature amount when the similarity calculation unit 116 calculates the similarity between the sample video and the detection target video. The attention level calculation unit 114 stores the calculated attention level in the storage unit 110 as attention level data 123.

[0048] The video search unit 115 controls a similarity search process that compares the detection target video data 131 with the sample video data 121 and searches the detection target video data for a video similar to the video of the sample video data in a section where the time-space instruction input is performed. This similarity search process searches the detection target video data for a video in which a person is performing a predetermined action.

[0049] The similarity calculation unit 116 calculates the similarity between the detection target video and the sample video based on the motion feature amount under the control of the video search unit 115. In this calculation, the attention level indicated by the attention level data 123 is used as a weight for the motion feature amount. As a result, the motion feature amount of a feature point closer to the position of the time-space instruction input is weighted more heavily to calculate the similarity. The calculation result of the similarity is notified to the video search unit 115, which extracts a video in which a specific motion is being performed from the detection target video based on the notified calculation result.

[0050] The analysis unit 117 performs various analyses based on the search results of the video search unit 115. For example, the analysis unit 117 evaluates the attractiveness and perceived price of a product based on the number of times that the action of picking up the product or the action of checking the price of the product is detected.

[0051] FIG. 5 is an example of a flowchart showing the overall processing procedure of the action detection and analysis process performed by the information processing device. First, in steps S11 and S12, pre-processing for motion detection is performed.

[0052] [Step S11] A sample video of a person performing a specific action to be detected is captured by the camera 200, and data of the captured sample video is input to the video input unit 111. The sample video provides a sample image showing the action of the person to be detected, and is used as a search query in a similarity search, which will be described later.

[0053] The feature amount calculation unit 112 extracts feature points from each frame of the sample video and calculates a motion feature amount for each feature point. The data of the sample video is stored in the storage unit 110 as sample video data 121. At the same time, the calculated motion feature amount is stored in the storage unit 110 as motion feature amount data 122 in association with the sample video data 121.

[0054] [Step S12] The time-space batch input unit 113 plays and displays the sample video based on the sample video data 121. The time-space batch input unit 113 accepts a time-space instruction input from the operator while playing and displaying the sample video. This allows the time range in which a specific action is performed in the sample video and the positions of the body parts of the person making a characteristic movement to be specified all at once.

[0055] The attention level calculation unit 114 calculates the attention level for each feature point for each frame of the sample video image in a section where the time-space instruction input is being made, based on the pointing position in the time-space instruction input. The calculated attention level is stored in the storage unit 110 as attention level data 123.

[0056] Next, in steps S13 and S14, motion detection is performed from the moving image of the detection target. [Step S13] A detection target video that is to be a detection target for a specific action is captured by the camera 200, and data of the captured detection target video is input to the video input unit 111. As the detection target video, for example, a video obtained by capturing video of the inside of a store over a one-day period is used.

[0057] The feature amount calculation unit 112 extracts feature points from each frame of the detection target video and calculates a motion feature amount for each feature point. Data of the detection target video is stored in the storage unit 110 as detection target video data 131. At the same time, the calculated motion feature amount is stored in the storage unit 110 as motion feature amount data 132 in association with the detection target video data 131.

[0058] [Step S14] The similarity calculation unit 116 calculates a similarity between the detection target video and a video of a section of the sample video where a time-space instruction input is being performed, based on the motion feature amount. In this calculation, the attention level indicated by the attention level data 123 is used as a weight for the motion feature amount. The video search unit 115 performs a similarity search using the similarity calculation result to detect a video in which a specific motion is being performed from the detection target video.

[0059] [Step S15] The analysis unit 117 analyzes the results of the detection of specific actions by the video search unit 115. For example, the analysis unit 117 evaluates the attractiveness or perceived price of the product based on the number of times the action of picking up the product or the action of checking the price of the product is detected.

[0060] Each process shown in FIG. 5 will be described in detail below. <Calculation of motion features from sample video> FIG. 6 is a diagram for explaining motion feature amounts. Motion feature amounts are feature amounts that represent a person's motion, and are calculated for each feature point. Recent advances in machine learning, particularly deep learning, have made it practical to estimate the presence area (circumscribing rectangle) of a person in an image, and the positions of body parts such as joints.

[0061] Therefore, in this embodiment, body parts of a person are used as feature points. For example, a plurality of predetermined body parts, such as both ends of the eyes, both shoulders, both wrists, the tips of ten fingers, and both toes, are used as feature points. Furthermore, a predetermined feature point number is assigned to each feature point. The body parts of a person are detected using a known skeleton detection technique.

[0062] If the feature point number is i and the number of detectable feature points (number of body parts) is n, then the motion feature amount P for a series of moving images is defined as a function of time t as shown in the following equation (1). P={p i (t)}(where i = 1 to n) (1) That is, the motion feature P is expressed by n functions p i As an example, Fig. 6 shows the motion feature amount calculated from the sample video when n = 2 (when two feature points are detected). In Fig. 6, the shooting time of the frame from the beginning of the sample video is t 1 ,t 2 ,t 3 , . . . The motion feature amount data 122 is created as a table in which feature point numbers and times (frame identification information) are associated with each other, as shown in FIG.

[0063] In addition, each motion feature includes a spatial attribute and a value attribute. That is, the above function p i (t) is the spatial attribute X(p i (t)) and value attribute A(p i Among these, the spatial attribute indicates two-dimensional coordinates of a feature point (body part) in a frame. In addition, if the distance z from the camera 200 to the person can also be detected, the spatial attribute may indicate three-dimensional coordinates. The value attribute will be described later.

[0064] If multiple people appear in an image, the motion feature is calculated for each person. For example, the people detected from the video are given labels (e.g., numbers) in the order of appearance, and the motion feature P is calculated in association with the labels.

[0065] 7 is an example of a flowchart showing a process of calculating motion features from a sample video sequence, which corresponds to the process in step S11 in FIG. [Step S21] A sample video showing a person performing a specific action to be detected is input from the camera 200 to the video input unit 111 and passed to the feature calculation unit 112. The feature calculation unit 112 stores data of the sample video in the storage unit 110 as sample video data 121.

[0066] [Step S22] The feature calculation unit 112 selects one frame from the beginning of the sample video sequence. [Step S23] The feature amount calculation unit 112 extracts feature points from the selected frames. As described above, the positions of body parts on the person are extracted as feature points. Each of the extracted feature points is assigned a feature point number that is uniformly determined in advance for each body part.

[0067] [Step S24] The feature calculation unit 112 calculates a motion feature for each extracted feature point. A space attribute and a value attribute are calculated as the motion feature. The feature calculation unit 112 registers the calculated motion feature in the motion feature data 122 in the storage unit 110.

[0068] [Step S25] The feature amount calculation unit 112 determines whether processing has been completed for all frames in the sample video. If there are unprocessed frames, the process proceeds to step S22, where one unprocessed frame is selected from the top. On the other hand, if all frames have been processed, the process ends.

[0069] [First example of motion features] FIG. 8 is a diagram showing a first example of a motion feature amount. As described above, the spatial attribute included in the motion feature amount indicates the coordinates of a feature point. Similarly to the spatial attribute, the value attribute included in the motion feature amount may also indicate the coordinates of a feature point. In this case, the coordinates of the i-th feature point in the frame at time t are expressed as (u i (t),v i (t)), then the function p i (t) is expressed by the following formula (2-1), and the spatial attribute and value attribute are expressed by formulas (2-2) and (2-3), respectively. Note that Fig. 8 illustrates the case where n=2 (when two feature points are detected). p i (t)=(u i (t),v i (t)) (2-1) X(p i (t))=(u i (t),v i (t)) ···(2-2) A(p i (t))=(u i (t),v i (t)) ···(2-3) [Second example of motion features] The motion feature amount can include, as a value attribute, information indicating a change in the feature point over time. For example, the feature amount calculation unit 112 calculates the velocity vector (u i '(t),v i In this case, the function p i (t) is expressed by the following formula (3-1), and the spatial attribute and value attribute are expressed by formulas (3-2) and (3-3), respectively. p i (t)=(u i (t),v i (t),u i '(t),v i '(t)) (3-1) X(p i (t))=(u i (t),v i (t)) ···(3-2) A(p i (t))=(u i (t),v i (t),u i '(t),v i '(t)) ···(3-3) In addition, the value attribute A(p i (t)) is u i '(t),v i '(t) only, u i (t),v i (t) may not be included.

[0070] [Third example of motion features] 9 is a diagram showing a third example of the motion feature. Information using deep learning technology may be calculated as a value attribute of the motion feature. For example, image frames each including a predetermined number of types (e.g., 100 types) of motions are prepared as training data. Each training data is assigned a motion label that identifies the motion depicted.

[0071] 9, a neural network including an array of several convolutional layers 141-1, . . . , 141-n that perform convolution for each feature point individually and a fully connected layer 142 common to all feature points is used for learning. In this neural network, a fully connected layer 142 that outputs an action label is arranged after the final convolutional layers 143-1, . . . , 143-n for each feature point. Then, using this neural network and the above training data, learning is performed in advance so that an action label of a frame of interest is output when a predetermined number of frames (e.g., five frames) before and after the frame of interest in a video are input.

[0072] After such learning is completed, the fully connected layer 142 is removed as shown in the lower part of FIG. 9. In this state, the output of the final convolutional layers 143-1, . . . , 143-n is expected to appropriately express the motion of a person for each corresponding feature point. In other words, the final convolutional layers 143-1, . . . , 143-n output appropriate features for discriminating the motion of a person for the corresponding feature points. Therefore, the feature calculation unit 112 inputs a predetermined number of frames around the frame at time t to the post-learning arrays 141-1, . . . , 141-n. As a result, the value attribute A(p 1 (t)),···,A(p n (t)) is calculated.

[0073] [Fourth example of motion features] By simultaneously capturing images of a person from different angles using multiple cameras, the three-dimensional coordinates of feature points (body parts) can be calculated using the principle of stereo vision. In this case, the function p i (t) is expressed as three-dimensional information as shown in the following formula (4-1). When using the method described in the "First Example of Motion Features" above, the spatial attribute and value attribute are expressed as formulas (4-2) and (4-3), respectively. p i (t)=(u i (t),v i (t),w i(t)) ···(4-1) X(p i (t))=(u i (t),v i (t),w i (t)) ···(4-2) A(p i (t))=(u i (t),v i (t),w i (t)) ···(4-3) The value attribute may be calculated using the method of the above-mentioned "Second Example of Motion Feature Amount" or "Third Example of Motion Feature Amount".

[0074] <Time-space instruction input and attention calculation> The time-space collective input unit 113 plays and displays the sample video, and accepts time-space instruction input from the operator. This time-space instruction input operation is an input operation for detecting a specific movement from the sample video. The time-space instruction input is performed when a person in the sample video is performing the movement to be detected, by pointing out the position of the body part of the person that makes the characteristic movement in the movement. The instruction input continues while the movement is being performed, and ends when the movement ends. This allows the time range (start time and end time) in which the specific movement is performed in the sample video and the position of the body part of the person that makes the characteristic movement to be specified all at once.

[0075] The time-space instruction input can be done by a simple operation using a mouse. For example, as shown in the following Figure 10, the time-space instruction input is done by dragging the mouse. Fig. 10 is a diagram showing an example of a time-space instruction input. Fig. 10 illustrates an example of detecting the action of picking up a product. In such an action, it is estimated that the movement of the wrist and palm is characteristic. Therefore, when a person on the reproduced image is picking up a product, the operator performs a time-space instruction input so that the mouse cursor (pointer) 151 points to a position as close as possible to the wrist or palm of the person.

[0076] In the example of FIG. 10, the sample video is frame F. 1 ~F 200 The frame F 31 ~F 180 Assume that the action of picking up the product is performed in 150 frames of F. In this case, the operator watches the sample video and 31 In frame F, the operator presses the click button while pointing the mouse cursor 151 near the wrist or palm of the person, and starts the drag operation. After that, the operator continues to press the click button while pointing the mouse cursor 151 near the wrist or palm of the person, and continues the drag operation, until the above operation is completed. Then, the operator 180 When this is displayed, release the click button and end the drag operation.

[0077] By such an operation, the time-space collective input unit 113 detects the start time t start and end time t end Get the start time t start is the frame F where a particular action was started. 31 The information indicates the end time t end is the final frame F in which this action is performed. 180 In FIG. 10, the start time t start At time t 1 is obtained, and the end time t end At time t 150 Furthermore, the time-space batch input unit 113 acquires the start time t start From end time t end During this time, the position indicated by the mouse cursor 151 is acquired in each frame.

[0078] Let R be the information obtained by the spatiotemporal instruction operation in this way. R is the time interval [t start ,t end] to a non-negative function defined on the image frame. If this information is represented as R(x,y;t), then R(x,y;t) represents the attention level (degree of contribution to the action) for the position (x,y) on the image frame at time t, as will be described in detail later. Hereinafter, R(x,y;t) will be referred to as the "attention level."

[0079] The method of inputting a time-space instruction is not limited to the above-mentioned mouse drag operation. For example, start and the end time t end For example, in the case of FIG. 10, the mouse is clicked once in frame F 31 The click button is pressed once while pointing the vicinity of the person's wrist or palm with mouse cursor 151. From that time, the mouse cursor 151 points to the vicinity of the person's wrist or palm without pressing the click button. Then, at the end of the movement, the click button is pressed again. As a result, the position pointed to by mouse cursor 151 in each frame from the first button press to the last button press is recorded.

[0080] Even when the button is pressed twice in this way, the vicinity of the characteristic body part for the motion to be detected can be recorded by the simple operation of continuing to point to a position on the screen with the mouse cursor 151, just like in the case of using a mouse drag operation. Furthermore, together with such spatial information, the time information of the start time and end time of the motion can be recorded all at once by a simple operation.

[0081] FIG. 11 is a diagram for explaining the calculation process of the attention level. In FIG. 11, the designated position (two-dimensional coordinates) of the mouse cursor 151 at time t is expressed as (x 0 (t), y 0 (t)) and the sample video at time t 1 From time t 150 Assume that a time-space instruction input is made during the display of each frame up to the time t 1 , time t 2 ,...,time t150 For each frame of 1 ),R(x,y;t 2 ),...,R(x,y;t 150 ) is calculated and registered in the attention level data 123.

[0082] [First example of attention] As a first example, the attention level R(x, y; t) can be calculated by the following formula (5). R(x,y;t)=exp(-((xx 0 (t) 2 +(yy 0 (t) 2 ) / σ 2 ) ···(5) According to this formula (5), the closer a feature point is to the position pointed to by the mouse cursor 151, the larger the attention level value becomes. As will be described in detail later, the attention level is used as a weight when calculating the similarity between feature points in similarity search. The higher the attention level, that is, the closer the feature point is to the position pointed to by the time-space instruction input, the larger the weight is assigned, and the similarity for that feature point greatly contributes to determining whether or not a specific action has been performed.

[0083] In addition, in formula (5), σ is a positive constant for adjusting the spatial extent of the attention level. Depending on the value of σ, an instruction can be given to the information processing device 100 to distinguish whether the motion to be detected is a (characteristic) motion involving a wide range of the body (e.g., the whole body) or a motion involving a narrow range (e.g., the fingertips). For example, if the motion to be detected is a motion involving a narrow range of the body, a small value can be set as σ. This makes it possible to limit the range in which the motion feature amount of the feature point contributes greatly to the similarity calculation, centered on the indicated position.

[0084] [Second example of attention] When inputting a spatiotemporal instruction, the operator may use the numeric keypad of a keyboard or the like in parallel with the mouse dragging operation described above to vary the spatial extent of the attention level as the frames progress. In this case, the attention level R(x,y;t) is calculated by the following formula (6). R(x,y;t)=exp(-((xx 0 (t) 2 +(yy 0 (t) 2 ) / σ 2 (t)) ···(6) In equation (6), σ 2 (t) is a positive coefficient for adjusting the spatial extent of the attention level set over time. By making σ variable over time as in equation (6), it is possible to adaptively set whether a wide range of body movements or a narrow range of body movements are characteristic within the flow of body movements. As a result, the accuracy of similarity search can be improved.

[0085] As another method of using the coefficient σ in calculating the attention level, for example, a method of using separate coefficients σ for the x coordinate and the y coordinate may also be applied. 12 is a flowchart showing an example of a calculation process of an attention level according to a time-space instruction input. The process in FIG. 12 corresponds to the process in step S12 in FIG.

[0086] <Step S31> The time-space batch input unit 113 starts playing and displaying the sample video based on the sample video data 121. [Step S32] The time-space batch input unit 113 determines whether a time-space instruction input has been started for the currently displayed frame. If a time-space instruction input has been started, the process proceeds to step S33. On the other hand, if a time-space instruction input has not been started, the process of step S32 is executed for the next frame to be displayed.

[0087] [Step S33] The time-space collective input unit 113 receives the start time t start is recorded in the storage unit 110. [Step S34] The time-space batch input unit 113 records in the storage unit 110 the position (coordinates) indicated by the mouse cursor in the current frame.

[0088] [Step S35] The time-space batch input unit 113 determines whether the time-space instruction input has ended for the next displayed frame. If the time-space instruction input has ended, the process proceeds to step S36. On the other hand, if the time-space instruction input has not ended, the process of step S34 is executed for the current frame (the frame for which the end is to be determined).

[0089] [Step S36] The time-space collective input unit 113 inputs the time for the previous frame as the end time t end and record it in the storage unit 110. [Step S37] The attention level calculation unit 114 calculates the attention level of the sample video at the start time t start From end time t end The first frame is selected from unprocessed frames included in the video segment up to (hereinafter referred to as the "input video segment").

[0090] [Step S38] The attention level calculation unit 114 calculates the attention level R for each pixel in the selected frame based on the position of the mouse cursor recorded in step S34. The calculated attention level R is recorded in the attention level data 123.

[0091] [Step S39] The attention calculation unit 114 determines whether all frames in the input section video have been processed. If there are unprocessed frames, the process proceeds to step S37, where the next unprocessed frame is selected. On the other hand, if all frames have been processed, the process ends.

[0092] 5 may be executed in parallel with the process of step S12. Also, the process of step S12 may be executed before the calculation of the motion feature amount in step S11. In this case, the motion feature amount may be calculated only for each frame included in the input section video determined by the spatiotemporal instruction input in step S12, among the sample video.

[0093] The attention level R may be calculated and recorded not for all pixels in a frame but for each feature point included in the frame. The attention level R may be calculated during a similarity search, which will be described later.

[0094] <Calculation of motion features from detection target video> Fig. 13 is an example of a flowchart showing a calculation process of motion features from a detection target video sequence. This Fig. 13 corresponds to the process of step S13 in Fig. 5. As shown in Fig. 13, the calculation of motion features from a detection target video sequence is performed in the same process as the calculation from a sample video sequence.

[0095] [Step S41] A detection target video is input from the camera 200 to the video input unit 111 and passed to the feature amount calculation unit 112. The detection target video is, for example, a video shot over the course of one day. The feature amount calculation unit 112 stores data of the detection target video in the storage unit 110 as detection target video data 131.

[0096] [Step S42] The feature calculation unit 112 selects one frame from the beginning of the detection target moving image. [Step S43] The feature amount calculation unit 112 extracts feature points (body parts) from the selected frames. Each extracted feature point is assigned a feature point number that is uniformly determined in advance for each body part. The same feature point number is assigned to the same body part between the sample video and the detection target video.

[0097] [Step S44] The feature calculation unit 112 calculates a motion feature for each extracted feature point. A space attribute and a value attribute are calculated as the motion feature. The feature calculation unit 112 registers the calculated motion feature in the motion feature data 132 in the storage unit 110.

[0098] [Step S45] The feature calculation unit 112 determines whether processing has been completed for all frames in the detection target moving image. If there are unprocessed frames, the process proceeds to step S42, where one unprocessed frame is selected from the top. On the other hand, if all frames have been processed, the process ends.

[0099] <Similarities-based action detection> The similarity calculation unit 116 calculates a similarity between the detection target video and a video of a section of the sample video where a time-space instruction input is performed (input section video) based on the motion feature amount. The video search unit 115 detects an input section video in which a specific motion is performed from the detection target video by a similarity search using the similarity calculation result.

[0100] FIG. 14 is a diagram for explaining the similarity search process between the sample video and the detection target video. As described above, the input section video 161 is a video (start time t start From frame t to end time t end In the similarity search process, a partial video having the same length as the input section video 161 is extracted from the detection target video 162, and the extracted partial video is compared with the input section video 161 to determine whether the partial video is similar to the input section video 161.

[0101] The partial video images are extracted in order from the beginning of the detection target video image 162, sliding one frame at a time toward the end. FIG. 14 illustrates an example in which m partial video images are extracted from the detection target video image 162. In this case, partial video images 163-1, 163-2, . . . , 163-m are extracted from the beginning of the detection target video image 162. Then, a similarity is calculated between each of the partial video images 163-1, 163-2, . . . , 163-m and the input section video image 161 based on the motion feature amount. If the similarity is equal to or greater than a predetermined threshold, it is determined that the partial video image includes a person performing a specific motion.

[0102] Note that dissimilarity (dissimilarity) may be calculated instead of similarity between video images. If the similarity is Sim and the dissimilarity is D, then the dissimilarity D can be converted to the similarity Sim by inverting the sign of the dissimilarity D, for example, by Sim=-D.

[0103] 15 is a diagram for explaining the calculation of similarity between an input section video and a partial video. The similarity calculation unit 116 calculates the similarity between frames (inter-frame similarity) between the input section video extracted from the sample video and the partial video extracted from the detection target video, starting from the top. The similarity calculation unit 116 then calculates the similarity between the input section video and the partial video by adding up all the calculated inter-frame similarities.

[0104] 15, the input section video includes frames 171-1, 171-2, 171-3, ... from the beginning, and the partial video includes frames 172-1, 172-2, 172-3, ... from the beginning. In this case, the inter-frame similarity is calculated between frames 171-1 and 172-1, between frames 171-2 and 172-2, and between frames 171-3 and 172-3. In this manner, the inter-frame similarity is calculated for all combinations of frames.

[0105] Between frames of the input section video and partial video, motion feature amounts of the same feature points (i.e., the same body parts) are compared. The smaller the difference between the motion feature amounts, the higher the similarity is determined to be. Between frames, the similarities calculated for each feature point in this way are added together to calculate the inter-frame similarity. However, when the similarities calculated for each feature point are added together, a weight is assigned to the similarity depending on the position of the feature point, based on the attention level based on the aforementioned spatiotemporal instruction input.

[0106] For example, in Fig. 15, it is assumed that a pointed position 173-1 is pointed to by a time-space input instruction in frame 171-1 of the input section video. In this case, the closer to the pointed position 173-1 among the feature points on frame 171-1, the greater the weight is given to the feature points when calculating the similarity between frames 171-1 and 172-1. Also, in Fig. 15, it is assumed that a pointed position 173-2 is pointed to by a time-space input instruction in frame 171-2 of the input section video. In this case, the closer to the pointed position 173-2 among the feature points on frame 172-1, the greater the weight is given to the feature points when calculating the similarity between frames 171-2 and 172-2.

[0107] As described above, the attention level R is set to a larger value for pixels closer to the position pointed to by the time-space input instruction. When comparing the motion feature amount of a certain feature point between frames 171-1 and 172-1, the attention level R corresponding to the position of the feature point on frame 171-1 is used as a weight. Therefore, the closer this feature point is to the position pointed to by the time-space input instruction, the larger the weight is multiplied by which the similarity based on the motion feature point is multiplied.

[0108] Through such calculations, the more characteristic the feature points of a body part are in the motion of the detection target, the greater the contribution to the similarity calculation. Therefore, the similarity between videos is an index that is sensitive to the similarity or difference in the motion of the body part that should be noted in the motion of the detection target. As a result, motion detection is performed with an emphasis on the motion of that body part, and the accuracy of motion detection can be improved.

[0109] The structure of the human body is complex, and when the body is photographed, the appearance of the body in the photographed image changes in various ways. Even when a person performs a specific action to be detected, there is diversity in the appearance of the body as a whole. For example, when a person performs the action of picking up a product, the movements of the body parts near the hands will be similar, whereas the movements of the feet may differ greatly depending on the person and the situation. For this reason, when trying to detect the action of picking up a product from an image, information on feature points near the hands is highly useful for action detection, but information on feature points near the feet is not very useful for action detection.

[0110] As described above, in this embodiment, the closer the motion feature value of a feature point is to the pointing position by the time-space pointing input, the higher the contribution to the similarity calculation for motion detection. For example, as shown by the dotted arrow in FIG. 15, among the feature points in frame 171-1, the feature point corresponding to the wrist, for example, is close to pointing position 173-1, but the feature point corresponding to the ankle is far from pointing position 173-1. Therefore, a larger weight (attention level) is assigned to the motion feature value of the feature point corresponding to the wrist than to the motion feature value of the feature point corresponding to the ankle. As a result, in the similarity calculation, the contribution of the motion feature value of the feature point corresponding to the wrist is higher than that of the feature point corresponding to the ankle. As a result, even if there is a variety in the body movements, a specific motion can be detected with high accuracy.

[0111] As a method for detecting motion from an image, for example, there is a method in which images in which the position and type of motion to be detected are given are used as training data for learning, and the learning results are used to detect motion from an input image. In this method, many postures of a person are given in the training data, so that a specific motion can be detected with high accuracy even when there is diversity in body movements. However, there is a problem in that a large amount of training data must be prepared in advance to achieve high accuracy. In particular, the position and type of motion to be detected must be specified manually for many images (training data), and creating training data requires a huge amount of work.

[0112] Another possible method of detecting a motion is to use posture information of a person as a search query to search for images containing a specific motion from among a large number of images. However, as described above, the postures that a person can take for a specific motion are diverse and change over time, so it is not easy to specify posture information that represents the motion to be detected, and it is difficult to prevent the detection accuracy from deteriorating due to the diversity of postures.

[0113] To cope with such a problem, according to the present embodiment, the number of steps required to prepare information in advance is reduced. That is, a person performing a specific action is photographed, and while playing and displaying the photographed moving image, a region in which the action to be focused on is performed is designated by a simple operation using a mouse such as a mouse drag operation (time-space instruction input), and the information processing device 100 can calculate the attention level for similarity calculation.

[0114] Here, in the time-space instruction input, the section in which a specific movement is taking place can be easily identified from the captured video by specifying the start time and end time using the mouse button. This eliminates the need to analyze unnecessary frames in the video, improving processing efficiency. At the same time, the area in which the movement to be focused on is taking place can be easily specified by specifying the mouse cursor, so the operator does not need to have high skills in image analysis or the like. In other words, by simply performing such a simple pre-operation, a specific movement can be detected with high accuracy even when there is diversity in body movements. Therefore, highly accurate motion detection can be performed efficiently with little effort.

[0115] [First example of similarity calculation] Here, the motion feature of the i-th feature point included in the input section of the sample video is expressed as P={p i Let Q be the motion feature of the i-th feature point included in the partial video of the detection target video. i For the attention level R(x, y; t) according to the spatiotemporal instruction input, the dissimilarity level D(P, Q; R) is calculated by the following formula (7). D(P,Q;R)=Σ t Σi R(X(p i (t));t)|A(p i (t))-A(q i (t))| 2 (7) In equation (7), R(X(p i (t));t) indicates the attention level corresponding to the position of the i-th feature point in the frame at time t. Σ indicates that the sum is taken over the subscripted variable. The range of the variable time t is the time interval [t start ,t end According to this formula (7), the closer a feature point is to the position pointed to by the spatio-temporal pointing input in the input section video, the greater the attention degree is multiplied as a weight for the value attribute A of the motion feature amount.

[0116] [Second example of similarity calculation] In the above formula (7), the scale (magnitude) of the value attribute can affect the accuracy of the similarity determination. In order to correctly determine similar actions as similar even if the scale of the value attribute differs depending on the size of the body or the position on the image, for example, cosine similarity can be used as shown in the following formula (8-1). Sim(P,Q;R) = Cos(P R ,Q R )=Cos(P R Q R / |P R ||Q R |) ···(8-1) P R ={R(X(p i (t));t)A(p i (t))} ···(8-2) Q R ={R(X(p i (t));t)A(q i (t))} ···(8-3) where i is a value from 1 to n, and t is the start time t start From end time t end The value can be up to . In addition, {...} represents a vector whose elements (values ​​on individual coordinate axes) are the values ​​when the variables i and t are changed within this range. "P RQ R " denotes the dot product of two vectors, and "|P R ||Q R "|" indicates the product of the squared norm of two vectors (the square root of the sum of the squares of the elements).

[0117] [Third example of similarity calculation] In the first and second examples above, the value attribute of the motion feature is used as is. Here, if the value attribute has position information (if the value differs depending on the body position), the following considerations must be made.

[0118] (1) For actions that are strongly affected by the position on the image (such as removing clothes that are in a fixed position), the value attribute has position information (fixed offset). (2) For behaviors that depend on the relative position of the target object in the image (for example, when the placement of clothes varies from day to day), the position information of the value attribute is replaced with the displacement from the target object (object offset).

[0119] (3) For actions that are determined independently of the position on the image (such as falling to the floor), the position information of the value attribute is replaced with the displacement from the center of the body (body offset). As the similarity or dissimilarity taking into account the object offset or the body offset, for example, A(p i (t)) to A(p i (t))-obj(t) or A(p i (t))-A(p μ (t)) for the motion feature Q. Here, obj(t) indicates a vector with the same length as the value attribute, and the position of the target object is set for elements with position information, and 0 is set for other elements. Also, A(p μ (t)) denotes the average value of the value attribute at all feature points.

[0120] 16 is an example of a flowchart showing a motion detection process based on similarity, which corresponds to the process in step S14 in FIG. [Step S51] The video search unit 115 searches for an input section video (sample video with start time t start From end time t end A video segment having the same length as the video segment (video segment up to the beginning of the video segment) is extracted from the beginning.

[0121] [Step S52] The similarity calculation unit 116 selects frames from the beginning of each of the input section video and the partial video, thereby specifying the video to be compared in steps S53 to S55.

[0122] [Step S53] The similarity calculation unit 116 selects feature points from the selected frames of the input section video sequence. [Step S54] The similarity calculation unit 116 obtains a motion feature amount corresponding to the feature point selected from the selected frames of the input section video in step S53, a motion feature amount corresponding to the feature point with the same feature point number present in the selected frames of the partial video, and an attention level corresponding to the coordinates of the former feature point. The similarity calculation unit 116 calculates an inter-frame similarity for the selected feature point based on the obtained motion feature amounts and attention levels.

[0123] [Step S55] The similarity calculation unit 116 determines whether processing of all feature points included in the selected frame has been completed. If there are unprocessed feature points, the process proceeds to step S53, where one unprocessed feature point is selected. On the other hand, if all feature points have been processed, the process proceeds to step S56.

[0124] [Step S56] The similarity calculation unit 116 determines whether processing of all frames included in the input section video (or partial video) has been completed. If unprocessed frames remain, the process proceeds to step S52, where the first frame of the unprocessed frames is selected. On the other hand, if all frames have been processed, the process proceeds to step S57.

[0125] [Step S57] The similarity calculation unit 116 adds up all the inter-frame similarities calculated in step S54, and outputs the similarity between the input section video sequence and the partial video sequence. [Step S58] The video search unit 115 compares the output similarity with a predetermined determination threshold to determine whether the partial video contains a specific action. If the similarity is equal to or greater than the determination threshold, it is determined that the partial video contains a specific action.

[0126] [Step S59] The video search unit 115 determines whether processing has been completed for all partial video images that can be extracted from the detection target image. If there are unprocessed partial video images, the process proceeds to step S51, where the first frame of the partial video image is advanced by one, and the next partial video image is extracted from the detection target image. On the other hand, if all partial video images have been processed, the process ends.

[0127] As described above, in the process of Fig. 16, the difference (dissimilarity) between frames may be calculated instead of the similarity between frames. In this case, in step S58, the added difference between frames may be converted into a similarity, and the converted similarity may be compared with a threshold value.

[0128] <Analysis processing> As an example, it is assumed that the action of picking up a product and the action of checking the price of the product are detected from detection target images taken in one day in steps S13 and S14 of Fig. 5. It is also assumed that the detection of the action of picking up a product is performed for each product.

[0129] In this case, the analysis unit 117 evaluates the attractiveness of the product by, for example, comparing the number of times the product was picked up in a day with the average number of times other products were picked up in a day. The analysis unit 117 also evaluates the affordability of the product by comparing the number of times the price of the product was checked in a day with the number of times the product was sold in the same day.

[0130] Fig. 17 is a diagram showing an example of an analysis table. The analysis table 181 shown in Fig. 17 is table information in which data for analyzing a product is temporarily registered. In this analysis table 181, the number of detections, the reference number of detections, the ratio, and the analysis results are registered in association with each of the actions of "picking up a product" and "checking the price."

[0131] Regarding the action of picking up a product, the number of times the product was picked up in a day is registered in the detection count item. The reference count indicates the average number of times the product was picked up in a day, including other products. The ratio indicates the ratio of the detection count to the reference count. For example, if the ratio is 1.4 or more, the product is evaluated as being highly attractive, if the ratio is 0.8 or more but less than 1.4, the product is evaluated as being moderately attractive, and if the ratio is less than 0.8, the product is evaluated as being lowly attractive, and the evaluation results are registered in the analysis result item.

[0132] Additionally, with regard to the price checking operation, the number of times the price of the relevant product was checked during the day is registered in the detection result item. The number of times the product was sold on the same day is registered in the reference number item. The ratio indicates the ratio of the number of detections to the reference number. For example, if the ratio is 4 or more, the relevant product is evaluated as overpriced, if the ratio is 2 or more but less than 4, the price of the relevant product is evaluated as appropriate, and if the ratio is less than 2, the relevant product is evaluated as underpriced, and the evaluation results are registered in the analysis result item.

[0133] Third embodiment Next, as a third embodiment, a case where the processing of the information processing device 100 according to the second embodiment is partially modified will be described.

[0134] In the second embodiment, the search results may over-match the movements of the person in the sample video used as the query image, which may cause erroneous detection of the movement and decrease the detection accuracy. Therefore, in the third embodiment, the information processing device plays and displays each of the partial video images found by the similarity search, and has the operator confirm whether a specific movement is actually captured in each partial video image. Then, the information processing device performs learning to distinguish whether the video image includes a specific movement or not, using the confirmation result and each partial video image. Finally, the information processing device uses the learning result to re-determine whether each partial video image includes a specific movement. This improves the detection accuracy of the movement.

[0135] Fig. 18 shows an example of the configuration of processing functions included in an information processing device according to the third embodiment. In Fig. 18, the same components as those in Fig. 4 are denoted by the same reference numerals, and their description will be omitted. The information processing device 100a shown in Fig. 18 includes a correction instruction input unit 118 and a discrimination learning unit 119 in addition to the storage unit 110, video input unit 111, feature amount calculation unit 112, time-space batch input unit 113, attention calculation unit 114, video search unit 115, similarity calculation unit 116, and analysis unit 117 shown in Fig. 4.

[0136] The correction instruction input unit 118 plays and displays each of the partial videos determined to include a specific action by the similarity search based on the similarity by the video search unit 115, and accepts an input indicating whether each partial video actually includes the specific action. The discrimination learning unit 119 uses each of the partial videos and the content of the input accepted by the correction instruction input unit 118 as training data to perform learning to discriminate whether a video includes a specific action.

[0137] Fig. 19 is an example of a flowchart showing the overall processing procedure of the motion detection and analysis processing in the third embodiment. In Fig. 19, the same processes as those in Fig. 5 are given the same step numbers. In the third embodiment, the processes of steps S61 and S62 are executed between steps S14 and S15 in Fig. 5.

[0138] [Step S61] The correction instruction input unit 118 plays and displays each of the partial video images in which the specific action was detected in step S14, and receives input from the operator indicating whether the specific action is actually included in each partial video image. The discrimination learning unit 119 uses these partial video images and the content of the input received by the correction instruction input unit 118 (i.e., information indicating whether the partial video image contains the specific action) as training data to perform learning to discriminate whether the video image contains the specific action.

[0139] [Step S62] Using the results of the learning performed in step S61, the video search unit 115 re-determines whether or not the specific action is included in each partial video image in which the specific action was detected in step S14.

[0140] Thereafter, in step S15, various analyses are carried out based on the results of the determination in step S62. 20 is an example of a flowchart showing the procedure of the learning and re-determination process. Note that steps S71 to S74 correspond to step S61 in FIG. 19, and steps S75 to S77 correspond to step S62 in FIG.

[0141] [Step S71] The correction instruction input unit 118 selects one of the partial video images in which a specific action was detected in step S14 of FIG. 19, and plays back and displays the selected partial video image. [Step S72] The operator checks the partial video image being played back and displayed, and performs an operation to input whether or not a specific action is captured in the video image. The correction instruction input unit 118 accepts the input information.

[0142] For example, if a specific action is captured in the partial video image being played back and displayed, the operator performs an input operation to indicate that fact. For example, the operator performs an operation to check a check section provided on the display screen. This notifies the correction instruction input unit 118 that the specific action is captured, and the process proceeds to the next step S73. On the other hand, if the specific action is not captured in the partial video image, the operator performs an operation to play and display the next partial video image. This notifies the correction instruction input unit 118 that the specific action is not captured, and the process proceeds to the next step S73.

[0143] [Step S73] The correction instruction input unit 118 determines whether processing has been completed for all partial video images in which a specific action was detected in step S14. If there are unprocessed partial video images, the process proceeds to step S71, where one of the unprocessed partial video images is selected. On the other hand, if all the applicable partial video images have been processed, the process proceeds to step S74.

[0144] [Step S74] Using the partial video images in which the specific action was detected in step S14 and the input result in step S72 as training data, discrimination learning unit 119 performs learning to discriminate whether or not the specific action is captured in the video image. In this learning, of these partial video images, the motion feature sets of each partial video image in which it has been input that the specific action is captured are used as positive examples, and the motion feature sets of each of the other partial video images are used as negative examples.

[0145] For example, a given function y=f(x;θ) from the feature space to the real space [0,1] is prepared. The discrimination learning unit 119 adjusts an unknown parameter θ so as to output y=1 for a group of motion features belonging to positive examples and y=0 for a group of motion features belonging to negative examples. In this way, when a value attribute of a motion feature is input, the discrimination result of whether it is a positive example or a negative example is learned to be output.

[0146] The value attribute input during learning is, for example, {A(q i(t))}, where i takes values ​​from 1 to n, and t is the start time t start From end time t end In addition, {} represents a vector whose elements (values ​​on individual coordinate axes) are the values ​​obtained when the variable it is changed within this range.

[0147] The learning can be performed using a machine learning technique such as a support vector machine (SVM). For example, x·θ is used as the function f(x;θ), where "·" indicates the inner product of vectors.

[0148] It is also desirable to assign weights to the value attributes according to their degree of attention. In this case, the input for learning is {R(X(p i (t));t)A(q i (t))}. [Step S75] The video search unit 115 selects one of the partial video images in which the specific action was detected in step S14.

[0149] [Step S76] The video search unit 115 inputs the value attribute of the motion feature of the selected partial video into the function y=f(x;θ) that is the learning result, to determine whether the partial video contains a specific motion. At this time, instead of inputting the value attribute as is, a value attribute weighted according to the attention level may be input. Moreover, the value obtained as a result of learning is used as θ. Moreover, whether the partial video contains a specific motion is determined by whether the output value of y is equal to or greater than a predetermined threshold value.

[0150] [Step S77] The video search unit 115 determines whether processing has been completed for all partial video images in which the specific action was detected in step S14. If there are unprocessed partial video images, the process proceeds to step S75, where one of the unprocessed partial video images is selected. On the other hand, if all applicable partial video images have been processed, the process ends. Note that the processes of steps S75 to S77 may be modified so that the process of step S76 is performed on all partial video images, including images in which the specific action was not detected.

[0151] As described above, learning is performed using the judgment result of the presence or absence of a specific motion input by the operator, and the motion is re-judged using the learning result, thereby making it possible to reduce the probability of erroneous detection of the motion and improve detection accuracy. Note that the operator can visually check the accuracy of the judgment result using the learning result, and if the accuracy is insufficient, can execute learning again.

[0152] [Fourth embodiment] In the second and third embodiments, the process for the sample video and the process for detecting a partial video including a specific action from the detection target video are executed in the same device. However, these processes may be executed in separate devices.

[0153] Fig. 21 is a diagram showing a configuration example of an information processing system according to a fourth embodiment. The information processing system shown in Fig. 21 includes a server device 100b and an information processing device 100c. In Fig. 21, the same components as those in Fig. 4 are denoted by the same reference numerals, and the description thereof will be omitted.

[0154] The server device 100b has a processing function related to a sample video image, which is broadly classified as one of the processing functions of the information processing device 100 according to the second embodiment. The server device 100b has a storage unit 110a, a communication unit 191, a feature amount calculation unit 112a, a time-space batch input unit 113, and an attention level calculation unit 114.

[0155] In this server device 100b, a sample video captured by camera 200 is transmitted from information processing device 100c and received by communication unit 192. Communication unit 191 passes the received sample video to feature amount calculation unit 112a. Then, feature amount calculation unit 112a calculates motion feature amounts for each feature point from the sample video, similar to feature amount calculation unit 112 in Fig. 4. As a result, feature amount calculation unit 112a stores sample video data 121 and motion feature amount data 122 in storage unit 110a.

[0156] The time-space collective input unit 113 also plays and displays the sample video on the display device and accepts the time-space instruction input. The attention level calculation unit 114 calculates the attention level for each feature point based on the designated position for each frame of the sample video in the section where the time-space instruction input is made. As a result, attention level data 123 is stored in the storage unit 110a.

[0157] When the above process is completed, the communication unit 191 transmits the action feature amount data 122 and the attention level data 123 stored in the storage unit 110a to the information processing device 100c. On the other hand, the information processing device 100c has a processing function of detecting a partial video image including a specific action from a detection target video image, broadly classified as one of the processing functions of the information processing device 100 according to the second embodiment. The information processing device 100c has a storage unit 110, a video input unit 111, a communication unit 192, a feature amount calculation unit 112b, a video search unit 115, a similarity calculation unit 116, and an analysis unit 117.

[0158] In this information processing device 100c, when a sample moving image is captured by the camera 200, the sample moving image is input to the video input unit 111. Then, the communication unit 192 stores the data of the sample moving image in the storage unit 110 as sample moving image data 121, and transmits the sample moving image to the server device 100b. After that, when the action feature amount data 122 and attention level data 123 corresponding to the sample moving image are transmitted from the server device 100b, the communication unit 192 receives them and stores them in the storage unit 110.

[0159] On the other hand, when a detection target moving image is captured by camera 200, the detection target moving image is passed from video input unit 111 to feature amount calculation unit 112b. Similar to feature amount calculation unit 112 in FIG. 4, feature amount calculation unit 112b calculates a motion feature amount for each feature point from the detection target moving image. As a result, feature amount calculation unit 112b stores detection target moving image data 131 and motion feature amount data 132 in storage unit 110.

[0160] After this, the video search section 115, the similarity calculation section 116 and the analysis section 117 execute the same processes as those of the information processing device 100 in FIG. The information processing device 100c may further include a correction instruction input unit 118 and a discrimination learning unit 119 shown in FIG.

[0161] The processing functions of the devices (e.g., the device that realizes the functions of the action recognition system 1 and the information processing device 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0162] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and then sold. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0163] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from a portable recording medium and execute processing according to the program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network. [Explanation of symbols]

[0164] 1. Motion Recognition System 2 Input reception section 3 Judgment section 10a, 10b, 20a, 20b Frame 11a,11b Indication position 12a, 12b, 22a, 22b Feature points

Claims

1. an input receiving unit that plays and displays a first moving image in which a person performing a specific action is captured, and that continuously receives, during the playing and display of the first moving image, an instruction input that indicates a position related to the specific action for each frame included in the first moving image; a determination unit that calculates a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determines whether or not the specific action is included in the second video based on the calculated similarity, wherein, when calculating the similarity, the determination unit assigns a greater weight to the comparison result of feature amounts for feature points in the first feature point group that are closer to a position pointed to by the instruction input; A motion recognition system having the above configuration.

2. the input receiving unit plays and displays a third moving image including the first moving image, and extracts, from the third moving image, a section from a position where the instruction input is started to a position where the instruction input is ended during the playing and display, as the first moving image. The action recognition system according to claim 1 .

3. the input receiving unit records a position pointed to by the instruction input for each frame included in the first moving image; the determination unit performs a comparison of feature amounts between the second video and the first video for each frame, and when calculating the similarity, determines the weight based on a designated position corresponding to a frame for which feature amounts are to be compared, among designated positions recorded for each frame included in the first video; The action recognition system according to claim 1 or 2.

4. the input reception unit receives an input for adjusting a setting value over time while receiving the instruction input; when calculating the similarity, the determination unit changes the weight value according to a distance from the instruction position by the instruction input to each of two or more feature points included in the first feature point group for each frame according to the input setting value. The action recognition system according to claim 3.

5. The determination unit is comparing the first video with a plurality of the second video to determine whether or not the specific action is included in each of the plurality of the second video; playing and displaying a fourth video determined to include the specific action among the plurality of second videos, and receiving a determination input indicating whether or not the specific action is included for each of the fourth videos; using the fourth video and the content of the judgment input as training data, learning is performed to discriminate whether the specific action is included in the video; determining a video including the specific action from among the fourth video using a result of the learning; The action recognition system according to claim 1 .

6. each of the two or more feature points included in the first feature point group is a body part of a person detected in the first video image; The action recognition system according to claim 1 .

7. The instruction input is performed by a mouse drag operation. The action recognition system according to claim 1 .

8. The computer playing and displaying a first moving image captured of a person performing a specific action, and continuously receiving, during the playing and display of the first moving image, an instruction input for indicating a position related to the specific action for each frame included in the first moving image; calculating a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determining whether or not the second video includes the specific action based on the calculated similarity; Execute the process, In the calculation of the similarity, a feature point in the first feature point group that is closer to the pointing position by the pointing input is weighted more heavily in a comparison result of the feature amounts. Action recognition method.

9. On the computer, playing and displaying a first moving image captured of a person performing a specific action, and continuously receiving, during the playing and display of the first moving image, an instruction input for indicating a position related to the specific action for each frame included in the first moving image; calculating a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determining whether or not the second video includes the specific action based on the calculated similarity; Execute the process, In the calculation of the similarity, a feature point in the first feature point group that is closer to the pointing position by the pointing input is weighted more heavily in a comparison result of the feature amounts. Motion recognition program.

Images