Similar video search program, device, and method

Abstract

To retrieve a video in which actions of multiple persons are similar.SOLUTION: A similar video retrieval apparatus recognizes each of multiple persons, each of objects, and actions of the persons included in an input video, specifies a type of each action of the persons, classifies a person who performs person-to-person action and a person who is a subject of the person-to-person action into the same group (group A) and classifies persons who perform person-to-object action on the same object into the same group (group B), calculates a tendency feature quantity indicating tendencies of actions of multiple persons on the basis of the groups and the number of persons belonging to the group, calculates tendency similarity which is a similarity in tendency feature quantity between an input video and a target video, calculates group similarity based on action feature quantities of persons belonging to the groups of the same action type, calculates inter-video similarity based on the tendency similarity and the group similarity, and outputs, as a retrieval result, a target video having the inter-video similarity which is equal to or higher than a predetermined value.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The disclosed technology relates to a similar video search program, a similar video search device, and a similar video search method. [Background technology]

[0002] Conventionally, a single video is used as input to search for videos similar to the input video from a large amount of accumulated video archives. Methods for searching for similar videos include a method based on image similarity such as similar color distribution, and a method that uses object recognition to recognize objects and their categories in the input video and search for videos that show objects of the same category. There is also a method that uses behavior recognition to search for videos that show people behaving similarly to the people in the input video.

[0003] For example, a video classification device has been proposed that detects facial pose data from an input video, and classifies the scenes of the input video based on the similarity between the detected facial pose data and reference facial pose data for each of various scenes that serve as classification categories.

[0004] Also, for example, an information processing device has been proposed that extracts a plurality of feature amounts representing characteristics of one video content from the video content, associates the extracted feature amounts with a person or a background, and determines the degree of correspondence between the associated feature amounts and feature amounts in another video content. This information processing device infers a relationship between a scene in the one video content and a scene in the other video content based on a comparison result obtained by comparing a plurality of feature amounts between a scene in the one video content from which the feature amounts were extracted and a scene in the other video content.

[0005] Also, for example, a video providing device has been proposed that has a video storage unit that stores a plurality of motion images and feature data that characterizes each motion image. This video providing device acquires feature data from a user's body motion, compares the acquired feature data with feature data stored in the video storage unit, and acquires a motion image similar to the user's body motion from the video storage unit. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2006 / 025272 [Patent Document 2] JP 2011-188342 A [Patent Document 3] JP 2007-323268 A Summary of the Invention [Problem to be solved by the invention]

[0007] In a method of searching for similar videos by behavior recognition, a technique of recognizing actions such as gestures is known. However, a technique of recognizing a specific gesture is limited to recognizing the behavior of one person. There are cases where a video showing multiple people is input and a similar scene is searched for, rather than searching for a video in which one of the people has a similar behavior. In this case, when behavior recognition that focuses on the behavior of an individual, such as the above-mentioned gesture recognition technique, is used, it may not be possible to search for the desired similar video.

[0008] In one aspect, the disclosed technology aims to search for videos in which multiple people's actions are similar. [Means for solving the problem]

[0009] In one aspect, the disclosed technology recognizes each of a plurality of people included in a first video and recognizes an action performed by each of the plurality of people. The disclosed technology also classifies each of the plurality of people into one of a plurality of groups set for each type of action based on the action performed by each of the recognized plurality of people. The disclosed technology then identifies a second video similar to the first video from among the plurality of videos based on a first feature amount related to at least one of the number of the groups and the number of people belonging to each group. Effect of the Invention

[0010] One aspect is that it has the effect of making it possible to search for videos in which multiple people's actions are similar. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a functional block diagram of the similar video search device. [Diagram 2] FIG. 13 is a diagram showing an example of a result of recognition and classification. [Diagram 3] FIG. 13 is a diagram showing an example of a video DB. [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as a similar video search device. [Diagram 5] 13 is a flowchart illustrating an example of a similar video search process. [Figure 6] FIG. 2 is a diagram illustrating an outline of the process of a similar video search device. [Figure 7] 10A and 10B are diagrams illustrating examples of tendency features of an input video and a target video. [Figure 8] 11 is a diagram showing an example of trend similarity between an input video and a target video; FIG. [Figure 9] 13 is a diagram showing an example of an A group similarity between an input video and a target video. FIG. [Figure 10] 1 is a diagram showing an example of inter-image similarity between an input image and a target image; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0013] 1, the similar video search device 10 according to this embodiment functionally includes a recognition unit 12, a classification unit 14, and an identification unit 16. Furthermore, a video DB (database) 20 is stored in a predetermined storage area of ​​the similar video search device 10.

[0014] The recognition unit 12 acquires an input video input to the similar video search device 10, and recognizes each of a plurality of people included in the input video, as well as the actions performed by each of the plurality of people. Furthermore, the recognition unit 12 recognizes objects included in the input video. The input video is an example of a "first video" of the disclosed technology.

[0015] Specifically, the recognition unit 12 recognizes each person from the input video by using, for example, a person recognition model such as a neural network generated in advance by machine learning. The recognition unit 12 assigns identification information (hereinafter referred to as a "person ID") to each recognized person. The recognition unit 12 also recognizes each object from the input video by using, for example, an object recognition model such as a neural network generated in advance by machine learning. The objects here include various things other than people, such as stationary objects such as walls and pillars, moving objects such as vehicles, portable objects such as bags and cardboard boxes, and plants and animals. The recognition unit 12 assigns identification information (hereinafter referred to as an "object ID") to each recognized object.

[0016] The recognition unit 12 recognizes each motion of the person recognized from the input video by, for example, existing behavior recognition technology such as gesture recognition. The motions are, for example, punching, kicking, walking, carrying, etc. The recognition unit 12 also recognizes the person ID of the other person who is the target of the motion when the recognized motion is an interpersonal motion that is a direct motion against another person. The recognition unit 12 also recognizes the object ID of the object that is the target of the motion when the recognized motion is an object-directed motion that is a direct motion against an object. The recognition unit 12 associates the person ID, the motion recognized for the person, and the person ID or object ID of the target of the motion for each person included in the input video in a predetermined storage area. FIG. 2 shows an example in which the "person ID", "motion", and "motion target" for each person are stored in a table format. The "motion type" and "group" in FIG. 2 will be described later.

[0017] The classification unit 14 classifies each of the multiple people into one of multiple groups set for each type of motion, based on the motion of each of the multiple people recognized from the input video. In this embodiment, a case will be described in which the types of motion are the above-mentioned motions against a person (e.g., punching, kicking, etc.), motions against an object (e.g., carrying, etc.), and other motions without a target (e.g., walking, etc.). The classification unit 14 assigns identification information (hereinafter referred to as a "type ID") indicating the type of motion of the person to each person. Here, the type ID of the motion against a person is "A", the type ID of the motion against an object is "B", and the type ID of the other motions is "C".

[0018] Furthermore, the classification unit 14 classifies a person who is the subject of an interpersonal action and a person who is the target of the interpersonal action into the same group. Specifically, the classification unit 14 refers to the person ID of the person to which the type ID=A indicating an interpersonal action is assigned and stored in association with the person ID of the target, and identifies the relationship between the subject and the person who is the target, thereby grouping the people who perform the interpersonal action. For example, when a person S performs an interpersonal action with a person T as a target, and a person T performs an interpersonal action with a person U as a target, the classification unit 14 classifies the three people S, T, and U into the same group.

[0019] The classification unit 14 also classifies people who are actors of actions directed to the same object into the same group. Specifically, the classification unit 14 refers to the person ID of the person to whom type ID=B indicating the action directed to the object is assigned and stored in association with the object ID of the action target, and classifies people who are actors who have the same object ID into one group. Even if the object type is the same, people who perform actions directed to different objects are classified into different groups. For example, even if person P and person Q each perform an action directed to an object, that is, if the cardboard box carried by person P is different from the cardboard box carried by person Q, that is, if the object IDs are different, person P and person Q are classified into different groups.

[0020] The classification unit 14 also classifies people performing other actions into the same group. Specifically, the classification unit 14 classifies people to whom type ID=C indicating other actions has been assigned into one group. The classification unit 14 assigns group identification information (hereinafter referred to as "group ID") to each classified group. "Type of Action" in FIG. 2 is the type ID of the type of action of the person indicated by the corresponding person ID, and "Group" is the group ID of the group to which the person indicated by the corresponding person ID is classified. In the example of FIG. 2, the group to which a person with type ID X (X=A, B, C) belongs is represented as "xk" which is a combination of type ID x (x=a, b, c) in lower case and identification number k such as a serial number. Hereinafter, a type of action with type ID X is also referred to as "type of action X", a group of type X is also referred to as "X group", and a group with group ID xk is also referred to as "group xk".

[0021] The identification unit 16 calculates a tendency feature that indicates a rough tendency of behavior of a plurality of people included in the input video based on at least one of the number of groups classified by the classification unit 14 and the number of people belonging to each group. The tendency feature is an example of a "first feature" of the disclosed technology. Specifically, the identification unit 16 may calculate a tendency feature including a ratio of the number of people belonging to a group that performs interpersonal or object-directed behavior to the total number of people included in the input video. The identification unit 16 may also calculate a tendency feature including a ratio of the number of groups that perform interpersonal or object-directed behavior to the number of people belonging to the group that performs interpersonal or object-directed behavior.

[0022] For example, the specification unit 16 may calculate, as the tendency feature, a feature vector having the following values ​​(i) to (iv) as elements. (i) The ratio of the number of people belonging to group A to the total number of people in the input video (ii) The ratio of the number of people in group A to the number of people in group A. (iii) The ratio of the number of people belonging to group B to the total number of people in the input video. (iv) The ratio of the number of people in group B to the number of people in group B.

[0023] Based on the calculated tendency feature of the input video, the identification unit 16 identifies a video similar to the input video from among the multiple videos stored in the video DB 20. The video similar to the input video is an example of the “second video” of the disclosed technology.

[0024] In the video DB20, for example, as shown in Fig. 3, video identification information (hereinafter referred to as "video ID"), video data, a tendency feature amount, and a motion feature amount are stored in association with each other for each of a plurality of videos. The tendency feature amount is calculated in advance for each video by the same calculation method as that for the tendency feature amount of the input video described above. The details of the motion feature amount will be described later, but like the tendency feature amount, it is calculated in advance for each video by the calculation method described later.

[0025] Specifically, the determination unit 16 calculates a tendency similarity, which is a similarity between the tendency feature amount of a target video (hereinafter referred to as a "target video") to be determined as to its similarity to the input video, among a plurality of videos stored in the video DB 20. For example, the determination unit 16 may calculate a cosine similarity between feature vectors representing the tendency feature amount as the tendency similarity.

[0026] The identification unit 16 also calculates inter-group similarity based on motion feature amounts relating to the motions of people included in the same type of motion group. The motion feature amount is an example of the "second feature amount" of the disclosed technology. As the motion feature amount, for example, a feature amount calculated in a behavior recognition technology used to recognize the motion of each person, for example, a coordinate value of each joint obtained by gesture recognition, may be used. Specifically, the identification unit 16 calculates the similarity of the motion feature amount for all combinations of a person included in the A group in the input video and a person included in the A group in the target video. As the similarity amount, for example, the cosine similarity between the feature vectors representing the motion feature amount may be used as described above. Then, the identification unit 16 calculates the statistical values ​​such as the average and maximum value of each calculated motion feature amount as the inter-group similarity of the A group (hereinafter referred to as the "A group similarity"). The identification unit 16 also calculates the B group similarity and the C group similarity for each of the B group and the C group in the same manner. Then, the specification unit 16 calculates statistical values ​​such as the average and maximum value of the A group similarity, the B group similarity, and the C group similarity as the inter-group similarity between the input video and the target video.

[0027] The identification unit 16 also calculates an inter-image similarity by integrating the trend similarity and inter-group similarity between the input image and the target image. The inter-image similarity is an example of an "integrated similarity" in the disclosed technology. For example, the identification unit 16 calculates the sum, weighted sum, or product of the trend similarity and the inter-group similarity as the inter-image similarity. The identification unit 16 identifies, from among the multiple images stored in the image DB 20, images whose inter-image similarity is equal to or greater than a predetermined value, or a predetermined number of images ranked in order of highest inter-image similarity, as images similar to the input image. The identification unit 16 outputs the identified images as search results.

[0028] The similar video search device 10 may be realized by, for example, a computer 40 shown in Fig. 4. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input / output device 44 such as an input unit and a display unit, and an R / W (Read / Write) unit 45 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 46 that is connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to one another via a bus 47.

[0029] The storage unit 43 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 43 serving as a storage medium stores a similar video search program 50 for causing the computer 40 to function as the similar video search device 10. The similar video search program 50 has a recognition process 52, a classification process 54, and an identification process 56. The storage unit 43 also has an information storage area 60 in which information constituting the video DB 20 is stored.

[0030] The CPU 41 reads out the similar video retrieval program 50 from the storage unit 43, loads it in the memory 42, and sequentially executes the processes of the similar video retrieval program 50. The CPU 41 operates as the recognition unit 12 shown in FIG. 1 by executing the recognition process 52. The CPU 41 also operates as the classification unit 14 shown in FIG. 1 by executing the classification process 54. The CPU 41 also operates as the identification unit 16 shown in FIG. 1 by executing the identification process 56. The CPU 41 also reads out information from the information storage area 60, and loads the video DB 20 in the memory 42. In this way, the computer 40 that has executed the similar video retrieval program 50 functions as the similar video retrieval device 10. The CPU 41 that executes the program is hardware.

[0031] The functions realized by the similar video search program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit).

[0032] Next, the operation of the similar video search device 10 according to this embodiment will be described. When an input video is input to the similar video search device 10 and a search for a similar video is instructed, the similar video search device 10 executes a similar video search process shown in Fig. 5. Note that the similar video search process is an example of a similar video search method of the disclosed technology.

[0033] In step S10, the recognition unit 12 acquires an input video input to the similar video search device 10. Next, in step S12, the recognition unit 12 recognizes each of the multiple people, each of the objects, and the actions performed by each of the multiple people included in the acquired input video. Then, for each person included in the input video, the recognition unit 12 associates a person ID, the action recognized for that person, and, if the recognized action is an action against a person or an action against an object, the person ID or object ID of the target of the action, and stores them in a specified storage area.

[0034] Next, in step S14, the classification unit 14 identifies the type of each of the actions of the multiple people based on the actions of each of the multiple people recognized from the input video, and stores a type ID (A, B, or C) indicating the type of action in association with the person ID. Furthermore, as shown in E of FIG. 6, the classification unit 14 classifies a person who is the performer of an action against a person and a person who is the target of the action against the person into the same group (group A), and classifies a person who is the performer of an action against an object against the same object into the same group (group B). Furthermore, the classification unit 14 classifies people performing other actions into the same group. Then, the classification unit 14 assigns a group ID to each classified group, and stores the group ID of the group to which each person belongs in association with the person ID.

[0035] Next, in step S16, the identification unit 16 calculates, for example, a feature vector having the above values ​​(i) to (iv) as elements as a tendency feature for the input video. Next, in step S18, the identification unit 16 selects one target video from the multiple videos stored in the video DB 20. Next, in step S20, as shown in F of Fig. 6, the identification unit 16 calculates a tendency similarity that is the similarity between the tendency feature of the input video calculated in the above step S16 and the tendency feature of the target video stored in the video DB 20 in association with the target video.

[0036] Next, in step S22, the determination unit 16 calculates the similarity of the motion feature for all combinations of people included in group A in the input video and people included in group A in the target video. Then, as shown in G of Fig. 6, the determination unit 16 calculates a statistical value (e.g., the maximum value) of each calculated motion feature as the A group similarity, and similarly calculates the B group similarity and the C group similarity. Then, the determination unit 16 calculates the statistical value (e.g., the maximum value) of the A group similarity, the B group similarity, and the C group similarity as the inter-group similarity between the input video and the target video.

[0037] Next, in step S24, the identification unit 16 calculates an integrated inter-video similarity by, for example, multiplying the trend similarity and inter-group similarity between the input video and the target video, as shown in H of Fig. 6. Next, in step S26, the identification unit 16 determines whether the inter-video similarity is equal to or greater than a predetermined value. If the inter-video similarity is equal to or greater than the predetermined value, the process proceeds to step S28, where the identification unit 16 adds the target video selected in step S18 above to a search result list, and the process proceeds to step S32.

[0038] On the other hand, if the inter-video similarity is less than the predetermined value, the process proceeds to step S30, where the identification unit 16 determines whether or not the next target video exists in the video DB 20. If the target video exists, the process returns to step S18, and if not, the process proceeds to step S32. In step S32, the identification unit 16 generates search results by, for example, sorting the videos included in the search result list in descending order of the inter-video similarity calculated for each video, outputs the generated search results, and the similar video search process ends.

[0039] Next, an example of determining similarity between an input video and a target video in the similar video search device 10 according to this embodiment will be described based on a specific example.

[0040] The input video is a video of a street brawl in which 10 people are fighting each other, and 3 passersby who are not involved in the brawl are also shown. The target video 1 is a video of a boxing match in which 2 fighters are fighting each other in the ring, and other people including a referee and 2 seconds are also shown in addition to the two people in the ring. The target video 2 is a video of a training scene at a boxing gym in which 2 trainees are sparring in the ring, and 10 other trainees and a coach are also shown in addition to the two people in the ring. The target video 3 is a video of a street brawl in which 12 people are fighting each other, and 3 passersby who are not involved are also shown.

[0041] In recognizing people and objects, multiple people are recognized from both images. Furthermore, from target image 2, punching bags, training equipment, etc. are also recognized as objects targeted by object-directed actions. Furthermore, in recognizing people's actions, the anti-personal action of "hitting someone" is recognized from both images. Furthermore, from target image 2, the anti-object action of "operating training equipment" is also recognized.

[0042] After the process of classifying people into groups based on the type of their actions has been completed, it is assumed that the actions, action target, type of action, and group for each person have been obtained for the input video, as shown in Fig. 2. For example, assume that a person with a person ID i1 (hereinafter referred to as "person i1"; the same applies to other person IDs) and person i2 are fighting each other. In this case, as shown in Fig. 2, a group a1 is formed of people i1 and i2. Also, assume that people i3 and i4 attack person i5, and person i5 fights back against person i4. In this case, as shown in Fig. 2, a group a2 is formed of people i3, i4, and i5.

[0043] FIG. 7 shows an example of the tendency feature of each video. The example of FIG. 7 shows a case where a tendency report quantity expressed by a feature vector having the above (i) to (iv) as elements is calculated. FIG. 8 shows the tendency similarity between an input video and each target video calculated using the tendency feature shown in FIG. 7. FIG. 8 is an example where the cosine similarity between the tendency feature quantities is calculated as the tendency similarity. For example, the tendency similarity between an input video and target video 1 is calculated as follows using the tendency feature quantities shown in FIG. 7.

[0044] Trend similarity (input video, target video 1) =(10 / 13*2 / 5+4 / 10*1 / 2+0*0+0*0) / {sqrt((10 / 13)^2+(4 / 10)^2+0^2+0^2)*sqrt((2 / 5)^2+(1 / 2)^2+0^2+0^2)} =0.91

[0045] When evaluating the similarity of actions between people belonging to group A, a similar action ("punching") is observed in all three target videos. If the maximum value of the similarity of the action features between people included in group A is used as the A group similarity, the A group similarity is calculated as shown in FIG. 9, for example. If target video 2 contains a person whose actions are very similar to those of a person in the input video, the A group similarity of target video 2 will be extremely high. On the other hand, since the input video does not contain a person belonging to group B, the B group similarity will be 0.

[0046] FIG. 10 shows the inter-video similarity between the input video and each target video when the inter-group similarity is the maximum value of the inter-group similarity A and the inter-group similarity B, and the inter-video similarity is "inter-video similarity = inter-group similarity x trend similarity". When the similarity between videos is determined using the similarity of the actions of a specific person in the video without considering the actions of multiple people included in the video, the video most similar to the input video is determined to be the target video 2. On the other hand, according to this embodiment, the trend similarity based on the trend feature indicating the trend of the actions of multiple people included in the video is taken into account, so that the target video 3 can be determined to be the video most similar to the input video. This makes it possible to output a search result in which the target video 3 is ranked first.

[0047] As described above, the similar video search device according to this embodiment recognizes each of the multiple people and each of the objects included in the input video, and recognizes the actions performed by each of the multiple people. Then, based on the actions performed by each of the recognized multiple people, the similar video search device classifies each of the multiple people into one of multiple groups set for each type of action. Furthermore, based on a tendency feature related to at least one of the number of groups and the number of people belonging to each group, the similar video search device identifies videos similar to the input video from among the multiple videos, and outputs the identified videos as search results. This makes it possible to search for videos in which the actions of multiple people are similar.

[0048] In the above embodiment, the case where the inter-image similarity is calculated using all images stored in the image DB as the target images has been described, but the present invention is not limited to this. For example, images with a trend similarity with the input image equal to or greater than a predetermined value, or images with a top predetermined number of trend similarities, may be extracted as candidate images, and search results may be obtained from the candidate images based on the inter-group similarity or inter-image similarity. Also, for example, in cases where the accuracy of the similarity determination may be loose, search results may be obtained based only on the trend similarity with the input image.

[0049] The disclosed technology can be used, for example, when searching a video archive for reference footage to be used in television programs such as news, or when detecting crime scenes from surveillance camera footage.

[0050] In the above embodiment, the similar video search program is stored (installed) in the storage unit in advance, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a non-transitory storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0051] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0052] (Appendix 1) Recognizing each of a plurality of people included in a first video and recognizing an action performed by each of the plurality of people; classifying each of the plurality of persons into one of a plurality of groups set for each type of movement based on the recognized movement performed by each of the plurality of persons; Identifying a second image similar to the first image from among a plurality of images based on a first feature amount related to at least one of the number of groups and the number of people belonging to each group. A similar video search program that causes a computer to execute a process including the above.

[0053] (Appendix 2) Further recognizing an object included in the first image; The types of actions include interpersonal actions against a person and object actions against an object. 2. A similar video search program as described in appendix 1.

[0054] (Appendix 3) The process of classifying into one of the multiple groups includes classifying a person who is the performer of a human-to-person action and a person who is the target of the human-to-person action into the same group, and classifying people who are the performers of object-directed actions against the same object into the same group.

[0055] (Appendix 4) A similar video search program as described in Appendix 2 or Appendix 3, wherein the first feature includes at least one of the following: a ratio of the number of people belonging to a group performing the interpersonal movement or the object movement to the total number of people included in the first video; and a ratio of the number of groups performing the interpersonal movement or the object movement to the number of people belonging to the group performing the interpersonal movement or the object movement.

[0056] (Appendix 5) A similar video search program according to any one of Supplementary Note 1 to Supplementary Note 4, wherein the process of identifying the second video includes identifying the second video based on a similarity in a second feature relating to the movements of people included in a group of the same type of movement.

[0057] (Appendix 6) The process of identifying the second video includes identifying, from among the plurality of videos, a video having an integrated similarity obtained by combining the similarity of the first feature and the similarity of the second feature that is equal to or greater than a predetermined value, or identifying a predetermined number of videos in order of highest integrated similarity.

[0058] (Appendix 7) The process of identifying the second image includes: extracting, from the plurality of videos, videos having a similarity of the first feature amount equal to or greater than a predetermined value, or a predetermined number of videos having a high similarity of the first feature amount as candidate videos; Among the candidate videos, a video having a similarity of the second feature amount or an integrated similarity obtained by integrating the similarity of the first feature amount and the similarity of the second feature amount equal to or greater than a predetermined value, or a predetermined number of candidate videos ranked in order of the similarity of the second feature amount or the integrated similarity are identified as the second video. 6. The similar video search program according to claim 5,

[0059] (Appendix 8) a recognition unit that recognizes each of a plurality of people included in a first video and recognizes an action performed by each of the plurality of people; a classification unit that classifies each of the plurality of persons into one of a plurality of groups that are set for each type of movement based on the recognized movement performed by each of the plurality of persons; an identification unit that identifies a second video similar to the first video from among a plurality of videos based on a first feature amount related to at least one of the number of the groups and the number of people belonging to each of the groups; A similar video search device comprising:

[0060] (Appendix 9) The recognition unit further recognizes an object included in the first image, The types of actions include interpersonal actions against a person and object actions against an object. 9. A similar video search device as described in appendix 8.

[0061] (Appendix 10) A similar video search device as described in Appendix 9, wherein the classification unit classifies a person who is the performer of an interpersonal action and a person who is the target of the interpersonal action into the same group, and classifies people who are the performers of object-directed actions against the same object into the same group.

[0062] (Appendix 11) A similar video search device as described in Appendix 9 or Appendix 10, wherein the first feature amount includes at least one of the following: a ratio of the number of people belonging to a group performing the interpersonal movement or the object movement to the total number of people included in the first video; and a ratio of the number of groups performing the interpersonal movement or the object movement to the number of people belonging to the group performing the interpersonal movement or the object movement.

[0063] (Appendix 12) The similar video search device according to any one of Supplementary Note 8 to Supplementary Note 11, wherein the identification unit identifies the second video based on a similarity between second features relating to the movements of people included in a group of the same type of movement.

[0064] (Appendix 13) The similar video search device described in Appendix 12, wherein the identification unit identifies, from the plurality of videos, videos for which an integrated similarity obtained by combining the similarity of the first feature and the similarity of the second feature is equal to or greater than a predetermined value, or a top predetermined number of videos in order of highest integrated similarity.

[0065] (Appendix 14) The identification unit is extracting, from the plurality of videos, videos having a similarity of the first feature amount equal to or greater than a predetermined value, or a predetermined number of videos having a high similarity of the first feature amount as candidate videos; Among the candidate videos, a video having a similarity of the second feature amount or an integrated similarity obtained by integrating the similarity of the first feature amount and the similarity of the second feature amount equal to or greater than a predetermined value, or a predetermined number of candidate videos ranked in order of the similarity of the second feature amount or the integrated similarity are identified as the second video. 13. A similar video search device as described in appendix 12.

[0066] (Appendix 15) Recognizing each of a plurality of people included in a first video and recognizing an action performed by each of the plurality of people; classifying each of the plurality of persons into one of a plurality of groups set for each type of movement based on the recognized movement performed by each of the plurality of persons; Identifying a second image similar to the first image from among a plurality of images based on a first feature amount related to at least one of the number of groups and the number of people belonging to each group. The similar video search method includes a process performed by a computer.

[0067] (Appendix 16) Further recognizing an object included in the first image; The types of actions include interpersonal actions against a person and object actions against an object. 16. The similar video search method according to claim 15.

[0068] (Appendix 17) The process of classifying into one of the multiple groups includes classifying a person who is the performer of an interpersonal action and a person who is the target of the interpersonal action into the same group, and classifying people who are the performers of object-directed actions against the same object into the same group.

[0069] (Appendix 18) A similar video search method as described in Appendix 16 or Appendix 17, wherein the first feature includes at least one of the following: a ratio of the number of people belonging to a group performing the interpersonal movement or the object movement to the total number of people included in the first video; and a ratio of the number of groups performing the interpersonal movement or the object movement to the number of people belonging to the group performing the interpersonal movement or the object movement.

[0070] (Appendix 19) A similar video search method described in any one of Supplementary Note 15 to Supplementary Note 18, wherein the process of identifying the second video includes identifying the second video based on a similarity in a second feature relating to the movements of people included in a group of the same type of movement.

[0071] (Appendix 20) Recognizing each of a plurality of people included in a first video and recognizing an action performed by each of the plurality of people; classifying each of the plurality of persons into one of a plurality of groups set for each type of movement based on the recognized movement performed by each of the plurality of persons; Identifying a second image similar to the first image from among a plurality of images based on a first feature amount related to at least one of the number of groups and the number of people belonging to each group. A non-transitory storage medium storing a similar video search program that causes a computer to execute a process including the steps of: [Explanation of symbols]

[0072] 10. Similar video search device 12 Recognition part 14 Classification section 16 Specific section 20 Video Database 40 Computer 41 CPU 42 Memory 43 Storage section 49 Storage medium 50 Similar video search program

Claims

1. Recognizing each of a plurality of people included in a first video and recognizing an action performed by each of the plurality of people; classifying each of the plurality of persons into one of a plurality of groups set for each type of movement based on the recognized movement performed by each of the plurality of persons; Identifying a second image similar to the first image from among a plurality of images based on a first feature amount related to at least one of the number of groups and the number of people belonging to each group. A similar video search program that causes a computer to execute a process including the above.

2. Further recognizing an object included in the first image; The types of actions include interpersonal actions against a person and object actions against an object. The similar video search program according to claim 1 .

3. The similar video search program of claim 2, wherein the process of classifying into one of the multiple groups includes classifying a person who is the subject of a human action and a person who is the target of the human action into the same group, and classifying people who are the subject of an object action against the same object into the same group.

4. A similar video search program as described in claim 2 or claim 3, wherein the first feature amount includes at least one of the following: a ratio of the number of people belonging to a group performing the interpersonal movement or the object movement to the total number of people included in the first video; and a ratio of the number of groups performing the interpersonal movement or the object movement to the number of people belonging to the group performing the interpersonal movement or the object movement.

5. A similar video search program according to any one of claims 1 to 4, wherein the process of identifying the second video includes identifying the second video based on the similarity of a second feature relating to the movements of people included in a group of the same type of movement.

6. The similar video search program of claim 5, wherein the process of identifying the second video includes identifying from among the plurality of videos a video having an integrated similarity obtained by combining the similarity of the first feature and the similarity of the second feature that is equal to or greater than a predetermined value, or a top predetermined number of videos in order of highest integrated similarity.

7. The process of identifying the second image includes: extracting, from the plurality of videos, videos having a similarity of the first feature amount equal to or greater than a predetermined value, or a predetermined number of videos having a high similarity of the first feature amount as candidate videos; Among the candidate videos, a video having a similarity of the second feature amount or an integrated similarity obtained by integrating the similarity of the first feature amount and the similarity of the second feature amount equal to or greater than a predetermined value, or a predetermined number of candidate videos ranked in order of the similarity of the second feature amount or the integrated similarity are identified as the second video. The similar video search program according to claim 5, further comprising:

8. a recognition unit that recognizes each of a plurality of people included in a first video and recognizes an action performed by each of the plurality of people; a classification unit that classifies each of the plurality of persons into one of a plurality of groups that are set for each type of movement based on the recognized movement performed by each of the plurality of persons; an identification unit that identifies a second video similar to the first video from among a plurality of videos based on a first feature amount related to at least one of the number of the groups and the number of people belonging to each of the groups; A similar video search device comprising:

9. Recognizing each of a plurality of people included in a first video and recognizing an action performed by each of the plurality of people; classifying each of the plurality of persons into one of a plurality of groups set for each type of movement based on the recognized movement performed by each of the plurality of persons; Identifying a second image similar to the first image from among a plurality of images based on a first feature amount related to at least one of the number of groups and the number of people belonging to each group. The similar video search method includes a process performed by a computer.

Images