Information processing device, information processing method, and recording medium

Abstract

This information processing device is provided with: a first selection unit that selects, from a first video, first to Nth first comparison frames where N is an integer of 2 or more; and a second selection unit that selects, from a second video, first to Nth second comparison frames respectively corresponding to the first to Nth first comparison frames. The order of the first to Nth first comparison frames follows the chronological order in the first video. The order of the first to Nth second comparison frames follows the chronological order in the second video. Each of the first to Nth second comparison frames is a plurality of frames indicating the same operation state as the corresponding one of the first to Nth first comparison frames.

Description

Information processing device, information processing method, and recording medium

[0001] The present invention relates to an information processing device, an information processing method, and a recording medium.

[0002] For example, Patent Document 1 discloses a technique for extracting skeletal information of each movement from a model video and an imitation video, and evaluating the similarity of each movement based on the similarity of each skeletal information at a synchronized time. The procedure for evaluating the similarity of each movement described in Patent Document 1 is roughly as follows.

[0003] According to Patent Document 1, when a user accesses and selects a model video from a server 3 from a user terminal 2, the model video is distributed from the server 3 to the user terminal 2. When playback of the model video Im begins, the camera of the user terminal 2 starts filming. Then, an imitation video Is of the user imitating the model video Im is recorded, and the completed filming of the imitation video Is is uploaded from the user terminal 2 to the server 3.

[0004] The server 3 estimates a person area from the model video Im and the imitation video Is, and extracts skeletal information from each person area. The server 3 calculates the amount of movement P of each body part based on the skeletal information extracted from the model video Im, and selects a comparison body part based on the calculation result of the amount of movement P.

[0005] The server 3 sets a similarity range E for each selected comparison part, and compares the skeletal information of the comparison part in the model video Im with the skeletal information of the corresponding part in the imitation video Is. The server 3 calculates a similarity score between the model's movement and the imitation's movement based on whether or not each corresponding part in the imitation video Is is located within the similarity range of each comparison part in the model video Im.

[0006] Japanese Patent Application Laid-Open No. 2020-195648

[0007] However, in the technology described in Patent Document 1, in order to compare the model and imitation movements shown in the model video and imitation video, respectively, only the start times of each movement are synchronized, which may make it difficult to compare each movement.

[0008] For example, if the model movement and the imitation movement are movements in which both hands are lowered and then extended upward, the overall time length of the movement may differ from the start of the movement, when both hands are lowered and then started to be raised, to the end of the movement, when both hands are extended upward. In this way, when the overall time lengths of the model movement and the imitation movement differ, with the technology described in Patent Document 1, there is a possibility that after one movement has finished, there will be no frame image to compare with the frame image showing the other movement.

[0009] Furthermore, for example, if the model movement and the imitation movement are movements in which both hands are lowered and then extended upward, the timing at which both hands reach shoulder height may differ. In this way, if the timing at which similar body states are reached differs during the respective processes of the model movement and the imitation movement, the technology described in Patent Document 1 may result in significant differences in the body states shown in the compared frame images.

[0010] In this way, when the overall duration of the movements differs, or when the timing at which similar physical states are reached in each movement differs, the technology described in Patent Document 1 is unable to compare appropriate frame images, which may result in making it difficult to accurately compare the content of each video.

[0011] In view of the above-mentioned problems, an example of an object of the present invention is to provide an information processing device, an information processing method, a program, etc. for easily comparing the contents of different videos.

[0012] According to one aspect of the present invention, there is provided an information processing device comprising: a first selection means for selecting, from a first video, first to Nth first comparison frames, where N is an integer equal to or greater than 2; and a second selection means for selecting, from a second video, first to Nth second comparison frames corresponding to the first to Nth first comparison frames, respectively; the first to Nth first comparison frames are ordered in chronological order in the first video; the first to Nth second comparison frames are ordered in chronological order in the second video; and each of the first to Nth second comparison frames is a plurality of frames that show the same operating state as the corresponding first to Nth first comparison frames.

[0013] According to one aspect of the present invention, there is provided an information processing method including one or more computers selecting, from a first video, first to Nth first comparison frames, where N is an integer equal to or greater than 2, and selecting, from a second video, first to Nth second comparison frames corresponding to the first to Nth first comparison frames, respectively, wherein the first to Nth first comparison frames are ordered in chronological order in the first video, the first to Nth second comparison frames are ordered in chronological order in the second video, and each of the first to Nth second comparison frames is a plurality of frames that show the same operating state as the corresponding first to Nth first comparison frames.

[0014] According to one aspect of the present invention, there is provided a recording medium having recorded thereon a program for causing one or more computers to: select, from a first video, first through Nth first comparison frames, where N is an integer equal to or greater than 2; and select, from a second video, first through Nth second comparison frames corresponding to the first through Nth first comparison frames, respectively; so that the first through Nth first comparison frames are ordered in chronological order in the first video; so that the first through Nth second comparison frames are ordered in chronological order in the second video; and so that each of the first through Nth second comparison frames is a plurality of frames that show the same operating state as the corresponding first through Nth first comparison frames.

[0015] According to one aspect of the present invention, it becomes possible to easily compare the contents of different videos, such as when the overall length of the movement is different or when the timing at which similar physical states (same movement states) are reached during the movement is different.

[0016] 1 is a diagram illustrating an overview of an information processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an overview of information processing according to the present disclosure. FIG. 3 is a diagram illustrating an example of an information processing system according to the present disclosure. FIG. 4 is a diagram illustrating an example of the functional configuration of a video storage device according to the present disclosure. FIG. 5 is a diagram illustrating an example of the functional configuration of a terminal device according to the present disclosure. FIG. 6 is a diagram illustrating an example of the functional configuration of an information processing device according to the present disclosure. FIG. 7 is a diagram illustrating an example of first comparison frames FRA_1 to FRA_N and second comparison frames FRB_1 to FRB_N selected from a first video and a second video, respectively. FIG. 8 is a diagram illustrating an example of skeleton information according to the present disclosure. FIG. 9 is a diagram illustrating an example of the physical configuration of an information processing device according to the present disclosure. FIG. 10 is a diagram illustrating an example of the physical configuration of a terminal device according to the present disclosure. FIG. 11 is a flowchart illustrating an example of information processing according to the present disclosure. FIG. 12 is a diagram illustrating an example of a first selection process for selecting a first comparison frame FRA_i. FIG. 13 is a diagram illustrating an example of a second selection process for selecting a second comparison frame FRB_i corresponding to the first comparison frame FRA_i. FIG. 14 is a diagram illustrating an example of a comparison screen according to the present disclosure. FIG. 15 is an enlarged view of the display areas of display items P1, P3, and P5 related to a comparison frame group FG_j. FIG. 10 is a diagram showing another example of the second selection process for selecting the second comparative frame FRB_i corresponding to the first comparative frame FRA_i.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] [Outline of the embodiment] FIG. 1 is a diagram showing an overview of an information processing device 103 according to the present disclosure.

[0019] The information processing device 103 includes a first selection unit 132 and a second selection unit 134. The first selection unit 132 selects first to Nth first comparison frames from the first video, where N is an integer equal to or greater than 2. The second selection unit 134 selects first to Nth second comparison frames from the second video that correspond to the first to Nth first comparison frames, respectively.

[0020] The order of the first to Nth first comparison frames follows the chronological order in the first video. The order of the first to Nth second comparison frames follows the chronological order in the second video. Each of the first to Nth second comparison frames is a plurality of frames that show the same motion state as the corresponding first to Nth first comparison frames.

[0021] This information processing device 103 makes it possible to easily compare the contents of different videos, such as when the overall length of the movement is different or when the timing at which a similar physical state (the same movement state) is reached during the movement is different.

[0022] FIG. 2 is a flowchart showing an overview of information processing according to the present disclosure.

[0023] The first selection unit 132 selects first to Nth first comparison frames from the first video, where N is an integer equal to or greater than 2 (step S103). The second selection unit 134 selects first to Nth second comparison frames from the second video, corresponding to the first to Nth first comparison frames, respectively (step S104).

[0024] The order of the first to Nth first comparison frames follows the chronological order in the first video. The order of the first to Nth second comparison frames follows the chronological order in the second video. Each of the first to Nth second comparison frames is a plurality of frames that show the same motion state as the corresponding first to Nth first comparison frames.

[0025] This information processing makes it possible to easily compare the content of different videos, such as when the overall length of the movement is different or when the timing at which similar physical states (same movement states) are reached during the movement is different.

[0026] [Embodiment 1] (Configuration example of information processing system 100) Fig. 3 is a diagram showing an example of the information processing system 100. The information processing system 100 is a system for comparing a first video and a second video. The information processing system 100 includes, for example, a video storage device 101, a terminal device 102, and an information processing device 103.

[0027] The video storage device 101, the terminal device 102, and the information processing device 103 are connected to each other via a communication network NT, and transmit and receive information to and from each other via the communication network NT. The communication network NT may be configured as a wired network, a wireless network, or a combination of these, for example.

[0028] The configuration of the information processing system 100 is not limited to this.

[0029] The first video and the second video are examples of different videos. The first video and the second video are, for example, videos showing the same action performed by an object. However, at least one of the object performing the action and the timing is different between the first video and the second video. The object is, for example, a person, an animal, etc. In the embodiment, an example in which the object is a person will be described.

[0030] Examples of such first and second images include images showing a model movement and a trainee's movement imitating the model movement, such as the movement of an athlete, dancer, or highly skilled technician.

[0031] Another example of the first and second images is an image showing the same action in the past and present (for example, extending both arms upwards from a lowered position), such as before and after rehabilitation, treatment, etc.

[0032] (Example of functional configuration of video storage device 101) Fig. 4 is a diagram showing an example of the functional configuration of the video storage device 101. The video storage device 101 is a device for storing the first video. Functionally, the video storage device 101 includes, for example, a video storage unit 111 and a video communication control unit 112.

[0033] The video storage unit 111 is a storage unit for storing the first video. For example, the video storage unit 111 may store the first video in advance. The first video is, for example, a video showing a model movement that has been filmed in advance.

[0034] The video communication control unit 112 transmits and receives information to and from the terminal device 102, the information processing device 103, etc. via the network NT. For example, in response to a request, the video communication control unit 112 transmits the requested first video, list information of the first video stored in the video storage unit 111, etc.

[0035] The functions of the video storage device 101 may be provided by other devices such as the terminal device 102 and the information processing device 103 .

[0036] (Example of Functional Configuration of Terminal Device 102) Fig. 5 is a diagram showing an example of the functional configuration of the terminal device 102. The terminal device 102 is a device for capturing the second video. The terminal device 102 is also a device for displaying a comparison screen including information relating to a comparison between the actions shown in the first video and the actions shown in the second video. This comparison screen (information indicating the comparison screen) is generated by the information processing device 103, as will be described in detail below.

[0037] The terminal device 102 functionally includes, for example, an input unit 121, an image capturing unit 122, a display unit 123, and a terminal communication control unit 124.

[0038] The input unit 121 accepts various inputs made by the user of the terminal device 102 .

[0039] For example, the input accepted by the input unit 121 may include a request for list information of the first video, an instruction to select the first video from the list information, a request to transmit the first video, an instruction to start and end shooting, and scrolling of the comparison screen.

[0040] For example, the input received by the input unit 121 may include a request for comparison of a first video and a second video (hereinafter also simply referred to as a "comparison request"). The first video related to the transmission or comparison request may be a first video selected from list information. Note that the input received by the input unit 121 is not limited to the examples given here, and may include at least one of these.

[0041] The image capturing unit 122 captures an image to generate a second video. For example, the image capturing unit 122 captures an image of the user of the terminal device 102 imitating an example of the user's actions in response to an instruction from the user. The second video thus generated shows the user imitating the example of the user's actions.

[0042] The display unit 123 displays various types of information.

[0043] The display unit 123 may, for example, display the video captured by the imaging unit 122. For example, the display unit 123 may display information received by the terminal communication control unit 124 (described later). This information may be, for example, a comparison screen, list information of the first video, or the first video selected with reference to the list information.

[0044] The terminal communication control unit 124 transmits and receives information to and from the video storage device 101, the information processing device 103, etc. via the network NT. For example, the terminal communication control unit 124 may transmit a request input by a user of the terminal device 102 to one or more devices such as the video storage device 101 and the information processing device 103. Furthermore, for example, the terminal communication control unit 124 may receive information transmitted from the one or more devices in response to the request.

[0045] The functions of the terminal device 102 may be provided by another device such as the video storage device 101 or the information processing device 103. Both the first video and the second video may be stored in advance in the video storage device 101, in which case there may be multiple video storage devices 101. Furthermore, both the first video and the second video may be captured by a device with a capture function, such as the terminal device 102, in which case there may be multiple terminal devices 102.

[0046] (Example of functional configuration of information processing device 103) Fig. 6 is a diagram showing an example of the functional configuration of the information processing device 103. The information processing device 103 is a device that performs processing for comparing a first video and a second video. Functionally, the information processing device 103 includes, for example, a first video acquisition unit 131, a first selection unit 132, a second video acquisition unit 133, a second selection unit 134, a similarity acquisition unit 135, and a display control unit 136.

[0047] The first video acquisition unit 131 acquires the first video. For example, the first video acquisition unit 131 acquires the first video related to the comparison request from the video storage device 101.

[0048] The first selection unit 132 selects the first to Nth first comparison frames from the first video. N is an integer equal to or greater than 2, and the same applies hereinafter. For example, the first selection unit 132 selects the first comparison frames from the first video acquired by the first video acquisition unit 131. Also, for example, the order of the first to Nth first comparison frames follows the chronological order in the first video.

[0049] The second video acquisition unit 133 acquires the second video. For example, the second video acquisition unit 133 acquires the comparison request and the second video related to the comparison request from the terminal device 102.

[0050] The second selection unit 134 selects the first to Nth second comparison frames from the second video. For example, the second selection unit 134 selects the second comparison frames from the second video acquired by the second video acquisition unit 133. The order of the first to Nth second comparison frames follows the chronological order of the second video. Each of the first to Nth second comparison frames is a plurality of frames that show the same operating state as the corresponding first to Nth first comparison frames.

[0051] Here, a frame is an image that constitutes a video, and is generally also referred to as a frame image. A frame that constitutes a first video is referred to as a first frame. A frame that constitutes a second video is referred to as a second frame. That is, the first frame and the second frame are arranged in chronological order to constitute the first video and the second video, respectively.

[0052] The first comparison frame and the second comparison frame are frames used to compare the first image and the second image, respectively, and are selected from the first frame and the second frame.

[0053] 7 is a diagram showing an example of first comparison frames FRA_1 to FRA_N and second comparison frames FRB_1 to FRB_N selected from the first and second images, respectively, where the fourth to (N-1)th first comparison frames FRA_4 to FRA_N-1 and second comparison frames FRB_4 to FRB_N-1 are omitted.

[0054] As shown in the figure, the first to Nth second comparison frames FRB_1 to FRB_N correspond to the first to Nth first comparison frames FRA_1 to FRA_N, respectively. The order of the first to Nth second comparison frames FRB_1 to FRB_N in the second video matches the order of the first to Nth first comparison frames FRA_1 to FRA_N in the first video. This order may, for example, follow the chronological order of the video.

[0055] Hereinafter, first comparison frames FRA_1 to FRA_N refer to the first to Nth first comparison frames, respectively, and second comparison frames FRB_1 to FRB_N refer to the first to Nth second comparison frames, respectively.

[0056] There are various methods for selecting the first comparative frames FRA_1 to FRA_N and the second comparative frames FRB_1 to FRB_N from the first and second images, respectively. In this embodiment, an example will be described in which the selection is made using the similarity of the frames.

[0057] The similarity obtaining unit 135 calculates the similarity between the frames to be compared. Hereinafter, the "similarity between the frames to be compared" will also be simply referred to as "similarity."

[0058] The similarity is, for example, a value indicating the degree of similarity between the motion states of a person shown in each of the compared frames. The motion state of a person is, for example, the posture of the person. The posture may be, for example, the posture of the entire body, or the posture of a predetermined or specified part of interest. The similarity of the posture of the entire body may be calculated, for example, based on the similarities of multiple predetermined parts.

[0059] In this embodiment, an example will be described in which the similarity between frames is calculated using skeletal information of a person shown in each of the frames to be compared. Note that the similarity is not limited to this, and may be, for example, the similarity between images shown in the entire frames to be compared, a predetermined region, etc.

[0060] In detail, for example, the similarity obtaining unit 135 functionally includes a skeleton information generating unit 135a and a skeleton similarity calculating unit 135b.

[0061] The skeletal information generator 135a generates skeletal information of the person shown in each of the frames to be compared. The skeletal information is an example of information indicating a person's posture, and is information indicating the state of the person's skeleton. The skeletal information is information indicating the posture of each predetermined part, such as the hip bones, spine, head, shoulders, left upper arm, left forearm, right upper arm, right forearm, left thigh, left lower leg, right thigh, and right lower leg. FIG. 8 is a diagram showing an example of skeletal information of a person with both hands raised.

[0062] A common technique may be used to generate such skeletal information. For example, the skeletal information generator 135a may input each of the frames to be compared into a learning model that, when an image is input, outputs skeletal information of the person shown in the image, thereby generating skeletal information of the person shown in each of the frames. This learning model may be, for example, a model trained using training data including learning images and skeletal information of the people shown in the images. The skeletal information shown here may be two-dimensional skeletal information on a screen or three-dimensional skeletal information in real space.

[0063] The skeleton similarity calculation unit 135b calculates the similarity between the frames to be compared using the skeleton information generated by the skeleton information generation unit 135a for each of the frames to be compared.

[0064] For example, the skeleton similarity calculation unit 135b calculates the similarity (part similarity) between the postures of each part indicated in the skeleton information, and calculates the overall similarity by integrating the part similarities for each frame.

[0065] The region similarity may be calculated using a predetermined relational expression based on, for example, the position of each region, the angle between regions, etc. The calculated region similarities for each region may then be integrated into an overall similarity by, for example, calculating the average region similarity, the maximum region similarity, or the sum of the region similarities. The average region similarity may be, for example, a simple average, or a weighted average of the region similarities using a weight predetermined for each region.

[0066] In this embodiment, an example will be described in which the overall similarity is adopted as the similarity between frames to be compared.

[0067] The similarity between frames is not limited to the overall similarity, but may be, for example, a region similarity regarding a region of interest.

[0068] The display control unit 136 causes the display unit 123 to display a comparison screen generated using information related to the comparison frame.

[0069] For example, the display control unit 136 generates a comparison screen using information related to the comparison frames. The information related to the comparison frames may include, for example, the frames to be compared, the similarity between the frames (at least one of overall similarity and local similarity), and at least one of skeletal information of the person shown in the frames. The display control unit 136 then transmits the comparison screen to the terminal device 102 via the network NT, causing the comparison screen to be displayed on the display unit 123.

[0070] (Example of Physical Configuration of Video Storage Device 101, Terminal Device 102, and Information Processing Device 103) Each of the video storage device 101, terminal device 102, and information processing device 103 is, for example, configured as a single physically different device.

[0071] Two or more of the video storage device 101, the terminal device 102, and the information processing device 103 may be physically configured as a single device. Also, one or more of the video storage device 101, the terminal device 102, and the information processing device 103 may be physically configured as multiple devices.

[0072] The video storage device 101 may have the same physical configuration as the information processing device 103. Therefore, the configuration of the information processing device 103 will be described here with reference to the drawings.

[0073] 9 is a diagram showing an example of the physical configuration of the information processing device 103. The information processing device 103 physically includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070.

[0074] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0075] The processor 1020 is implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0076] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0077] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the device that includes the storage device 1040. The processor 1020 loads each of these program modules into the memory 1030 and executes them to realize the function corresponding to that program module.

[0078] The network interface 1050 is an interface for connecting a device equipped with the network interface 1050 to the network NT.

[0079] The input interface 1060 is an interface for the user to input information, and is configured from, for example, a touch panel, a keyboard, a mouse, and the like.

[0080] The output interface 1070 is an interface for presenting information to the user, and is configured, for example, by a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like.

[0081] 10 is a diagram showing an example of the physical configuration of the terminal device 102. The terminal device 102 physically includes, for example, a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070, which are similar to those of the information processing device 103. The terminal device 102 further includes, for example, a camera 1080. The camera 1080 is connected to the bus 1010, captures an image of a subject, and generates a video including an image of the subject.

[0082] (Example of Operation of Information Processing System 100) The information processing device 103 executes information processing for comparing the first video and the second video. Fig. 11 is a flowchart showing an example of the information processing.

[0083] For example, the information processing is initiated by transmitting a comparison request together with the second video from the terminal communication control unit 124 in response to input received by the input unit 121. The second video is video generated by the imaging unit 122 capturing an image in response to a user's instruction. The second video is, for example, a video showing the user imitating an example. The comparison request may also include information (first video identification information) for identifying the first video to be compared with the second video. The first video identification information is, for example, information for the user of the terminal device 102 to refer to list information acquired from the video communication control unit 112 and identify the first video selected from the list information.

[0084] The trigger for starting information processing is not limited to this.

[0085] The second video acquisition unit 133 acquires a comparison request and a second video (step S101).

[0086] For example, the second video acquisition unit 133 acquires the comparison request and the second video transmitted together therewith from the terminal device 102 .

[0087] The first video image acquisition unit 131 acquires the first video image (step S102).

[0088] For example, the first video acquisition unit 131 transmits the first video identification information included in the comparison request acquired in step S101 to the video storage device 101. Upon acquiring the first video identification information, the video communication control unit 112 reads out the first video identified by the first video identification information from the video storage unit 111 and transmits it to the information processing device 103. The first video acquisition unit 131 acquires the transmitted first video.

[0089] It should be noted that the first video acquisition unit 131 and the second video acquisition unit 133 only need to acquire the first video and the second video to be compared, respectively, and the acquisition method is not limited to the example given here.

[0090] The first selection unit 132 selects first comparative frames FRA_1 to FRA_N from the first video acquired in step S102 (step S103). The order of the first comparative frames FRA_1 to FRA_N follows the chronological order in the first video.

[0091] (Example of Method for Selecting First Comparative Frame) For example, the first selection unit 132 selects a first comparative frame FRA_1 from the first video. The first comparative frame FRA_1 is, for example, the first frame in the first video that first shows a target person.

[0092] The method for selecting the first comparative frame FRA_1 is not limited to this. For example, the first comparative frame FRA_1 may be the first frame of the first video at a predetermined point in time.

[0093] When selecting the first comparison frame FRA_i, the first selection unit 132 selects from the first frames that are later than the first comparison frame FRA_i-1 in chronological order in the first video. Therefore, the order of the first comparison frames FRA_1 to FRA_N matches the chronological order of the first comparison frames FRA_1 to FRA_N in the first video. i is an integer between 2 and N, inclusive, and the same applies hereinafter.

[0094] When selecting the first comparison frame FRA_i, the first selection unit 132 uses the similarity between the first comparison frame FRA_i-1 and a first frame that comes after the first comparison frame FRA_i-1 in chronological order in the first video. For example, the first selection unit 132 selects as the first comparison frame FRA_i the first frame that is closest to the first comparison frame FRA_i-1 in chronological order in the first video and whose similarity to the first comparison frame FRA_i-1 is equal to or less than a third threshold.

[0095] This allows the first selection unit 132 to select, as the first comparison frame FRA_i, a first frame that is close to the first comparison frame FRA_i-1 but shows a different posture (motion state) from that of the first comparison frame FRA_i-1. In other words, it is possible to collect characteristic postures as the first comparison frame.

[0096] The third threshold may be a predetermined value, or may be a value that is appropriately set by a user of the terminal device 102 or the like. Furthermore, the third threshold may be adjusted so that the number N of first comparative frames FRA_1 to FRA_N selected from the first video becomes a predetermined reference number.

[0097] FIG. 12 is a diagram showing an example of the first selection process for selecting the first comparative frame FRA_i in step S103.

[0098] The similarity obtaining unit 135 obtains the similarity between the first comparative frame FRA_i-1 and the first frame of the selection candidate (step S103a).

[0099] The first frame of the selection candidate in the first step S103a is, for example, a frame that is a predetermined number (a predetermined number equal to or greater than 1) after the first comparative frame FRA_i-1 in chronological order in the first video.

[0100] The first selection unit 132 determines whether the similarity calculated in step S103a is equal to or less than a third threshold value (step S103b).

[0101] If the similarity is equal to or less than the third threshold (Step S103b; Yes), the first selection unit 132 selects the first frame of the selection candidate as the first comparative frame FRA_i (Step S103c).

[0102] If the similarity is not less than the third threshold (step S103b; No), the similarity acquisition unit 135 sets the first frame of the selection candidate to a frame that is a predetermined number (a predetermined number greater than or equal to 1) after the first frame in chronological order in the first video (step S103d).

[0103] In step S103d, the frame for which step S103b has been determined to be No is removed from the selection candidates for the first frame. Step S103a is executed again using the first frame of the selection candidates set here.

[0104] The first selection process may be repeatedly executed until no first frames having a value equal to or less than the third threshold are detected from the first video.

[0105] If a reference number is set in advance for the number N of first comparison frames FRA_1 to FRA_N, the third threshold may be changed so that the number N of first comparison frames FRA_1 to FRA_N becomes the reference number. For example, if the number N of selected first comparison frames FRA_1 to FRA_N is greater than the reference number, the third threshold may be decreased. Also, for example, if the number N of selected first comparison frames FRA_1 to FRA_N is less than the reference number, the third threshold may be increased. The first selection process may then be repeatedly executed until the number N of first comparison frames FRA_1 to FRA_N becomes the reference number. An upper limit on the number of repetitions may be set in advance.

[0106] Note that the method for selecting the first comparison frames FRA_1 to FRA_N is not limited to this. For example, the first comparison frames FRA_1 to FRA_N may be designated by a user of the terminal device 102. In this case, the first video identification information may further include information for identifying the designated first comparison frames FRA_1 to FRA_N.

[0107] Referring again to FIG. 11 , the second selection unit 134 selects second comparison frames FRB_1 to FRB_N from the second video acquired in step S101 (step S104). The order of the second comparison frames FRB_1 to FRB_N follows the chronological order in the second video. Each of the second comparison frames FRB_1 to FRB_N is a plurality of frames that show the same motion state as the corresponding first comparison frames FRA_1 to FRA_N.

[0108] (Example of Method for Selecting Second Comparison Frame) For example, the second selection unit 134 selects a second comparison frame FRB_1 from the second video. The second comparison frame FRB_1 is, for example, the first second frame in chronological order in the second video whose similarity to the first comparison frame FRA_1 is equal to or greater than a predetermined first threshold.

[0109] This allows a frame in which the motion state is the same as the start state of the motion shown in the first comparative frames FRA_1 to FRB_N to be selected as the second comparative frame FRB_1.

[0110] The method for selecting the second comparative frame FRB_1 is not limited to this.

[0111] When selecting the second comparison frame FRB_i, the second selection unit 134 selects from the second frames that are later than the second comparison frame FRB_i-1 in chronological order in the second video. Therefore, the order of the second comparison frames FRB_1 to FRB_N matches the chronological order of the second comparison frames FRB_1 to FRB_N in the second video.

[0112] This allows frames in chronological order in the second video to be selected as the second comparative frames FRB_1 to FRB_N.

[0113] As described above, the second comparison frames FRB_1 to FRB_N correspond to the first comparison frames FRA_1 to FRA_N, respectively. The order of the first comparison frames FRA_1 to FRA_N matches the chronological order of the first comparison frames FRA_1 to FRA_N in the first video. Therefore, the order of the second comparison frames FRB_1 to FRB_N in the second video matches the chronological order of the first comparison frames FRA_1 to FRA_N in the first video.

[0114] When selecting the second comparison frame FRB_i, the second selection unit 134 uses the similarity (hereinafter, this similarity is also referred to as the "first similarity") between the first comparison frame FRA_i and each of the second frames that come after the second comparison frame FRB_i-1 in chronological order in the second video.

[0115] For example, the second selection unit 134 selects the second comparison frame FRB_i from among the second frames that are later than the second comparison frame FRB_i-1 in chronological order in the second video, based on the first similarity. In detail, for example, the second selection unit 134 selects the second frame that has the largest first similarity as the second comparison frame FRB_i from among the second frames that are later than the second comparison frame FRB_i-1 in chronological order in the second video.

[0116] This allows the selection of the second frame FRB_i that shows the motion state most similar to (the same motion state as) the motion state shown in the first comparison frame FRA_i. Note that the "same motion state" is not limited to the motion state with the highest first similarity, but may be, for example, a motion state with the first similarity equal to or greater than a predetermined threshold.

[0117] FIG. 13 is a diagram showing an example of the second selection process for selecting the second comparative frame FRB_i corresponding to the first comparative frame FRA_i in step S104.

[0118] The similarity obtaining unit 135 obtains the similarity between the first comparison frame FRA_i and each of the second frames of the selection candidates (step S104a). This similarity is the first similarity.

[0119] For example, the second frame of the selection candidate may be all frames in the second video that follow the second comparative frame FRB_1.

[0120] For example, the second frame of the selection candidate may be a predetermined number of second frames after the second comparative frame FRB_1 in the second video. The predetermined number used in this second selection process may be the same as or different from the predetermined number used in the first selection process.

[0121] The second selection unit 134 determines whether or not any of the first similarities calculated in step S104a is equal to or greater than a predetermined first threshold value (step S104b).

[0122] If there is no first similarity that is equal to or greater than the first threshold (step S104b; No), the second selection unit 134 does not select the second comparison frame FRB_i and ends the second selection process. Therefore, if the first similarity is less than the first threshold for all or a predetermined number of second frames that are chronologically subsequent to the second comparison frame FRB_i-1 in the second video, the second comparison frame FRB_i does not need to be selected.

[0123] If the first similarity is equal to or greater than the first threshold (step S104b; Yes), the second selection unit 134 selects the second frame of the selection candidate having the highest first similarity as the second comparison frame FRB_i (step S104c), and terminates the second selection process. As a result, the second frame of the selection candidate having the highest first similarity, which is equal to or greater than the predetermined first threshold, is selected as the second comparison frame FRB_i.

[0124] This second selection process may be repeatedly executed until i reaches N, starting from 2, for example.

[0125] Note that the method for selecting the second comparison frames FRB_1 to FRB_N is not limited to this. For example, the second selection process does not need to include step S104b. In this case, the second frame with the highest first similarity among the second frames to be selected is selected as the second comparison frame FRB_i.

[0126] Referring again to Fig. 11, the display control unit 136 generates a comparison screen and displays it on the display unit 123 (step S105), and then ends the information processing.

[0127] For example, the display control unit 136 generates a comparison screen (information showing the comparison screen) in which information related to the comparison frames is arranged in chronological order, using the first comparison frames FRA_1 to FRA_N and the second comparison frames FRB_1 to FRB_N selected in steps S103 to S104, and the similarity determined in the selection in steps S103 to S104.

[0128] For example, the display control unit 136 transmits the generated comparison screen (information indicating the comparison screen) to the terminal device 102. When the terminal communication control unit 124 acquires the transmitted information, it causes the display unit 123 to display the comparison screen indicated by the information. As a result, the display control unit 136 causes the display unit 123 to display the comparison screen.

[0129] (Configuration Example of Comparison Screen) The comparison screen may include, for example, at least one of the following display items P1 to P5: Note that the display items included in the comparison screen are not limited to the following examples.

[0130] The display item P1 is an area that associates the corresponding first comparative frames FRA_1 to FRA_N and second comparative frames FRB_1 to FRB_N for each of at least one comparative frame group FG_1 to FG_N.

[0131] The comparison frame groups FG_1 to FG_N are each composed of corresponding first comparison frames FRA_1 to FRA_N and corresponding second comparison frames FRB_1 to FRB_N, respectively. That is, the comparison frame group FG_j is each composed of corresponding first comparison frames FRA_j and second comparison frames FRB_j, respectively, where j is an integer from 1 to N, inclusive, and the same applies hereinafter.

[0132] The display item P2 is an area showing at least one of the frame positions of the first comparative frames FRA_1 to FRA_N in the first video and the frame positions of the second comparative frames FRB_1 to FRB_N in the second video.

[0133] Display item P3 is the degree of similarity between a first comparative frame FRA_j and a second comparative frame FRB_j included in at least one comparative frame group FG_j.

[0134] Display item P4 is the action similarity, which is the similarity between the actions shown in the first video and the second video.

[0135] For example, the display control unit 136 calculates the action similarity based on the similarity of the actions shown in the corresponding first comparison frames FRA_1 to FRA_N and the corresponding second comparison frames FRB_1 to FRB_N. The action similarity may include, for example, at least one of the average, maximum, or the like of the similarity of the actions shown in the corresponding first comparison frames FRA_1 to FRA_N and the corresponding second comparison frames FRB_1 to FRB_N. Note that the action similarity may be any value that indicates the overall degree of similarity between the actions shown in the first comparison frames FRA_1 to FRA_N and the actions shown in the second comparison frames FRB_1 to FRB_N, and is not limited to the example given here.

[0136] Display item P5 is an area that associates, for each of at least one group of comparison frames FG_1 to FG_N, human skeletal information shown in each of the corresponding first comparison frames FRA_1 to FRA_N and second comparison frames FRB_1 to FRB_N.

[0137] Fig. 14 is a diagram showing an example of a comparison screen SC. The comparison screen SC shown in Fig. 14 is an example including display items P1, P3, and P5 relating to a plurality of comparison frame groups FG_1 to FG_N.

[0138] In FIG. 14, the display areas of the display items P1, P3, and P5 relating to the comparative frame groups FG_4 to FG_N-1 are omitted.

[0139] In the comparison screen SC, the comparison frame groups FG_1 to FG_N are arranged in the chronological order of the first comparison frames FRA_1 to FRA_N included in each of the comparison frame groups FG_1 to FG_N in the first video. That is, the comparison frame groups FG_1 to FG_N are arranged in the chronological order of the first comparison frames FRA_1 to FRA_N in the first video.

[0140] Furthermore, as described above, the chronological order of the second comparison frames FRB_1 to FRB_N in the second video matches the chronological order of the first comparison frames FRA_1 to FRA_N in the first video, so it can be said that the comparison frame group FG_1 to FG_N is arranged in the chronological order of the second comparison frames FRB_1 to FRB_N in the second video.

[0141] The display areas of the display items P1, P3, and P5 relating to the comparative frame groups FG_1 to FG_N may be configured similarly.

[0142] FIG. 15 is an enlarged view showing the display areas of the display items P1, P3, and P5 related to the comparative frame group FG_j.

[0143] As shown in FIG. 15, the display area of ​​the display item P1 related to the comparative frame group FG_j includes each of the first comparative frame FRA_j and the second comparative frame FRB_j.

[0144] Furthermore, in the display area of ​​the display item P5 for the comparison frame group FG_j, the skeletal information of the person shown in each of the first comparison frame FRA_j and the second comparison frame FRB_j is superimposed and associated with each other. In the example shown in the same figure, the skeletal information based on the first comparison frame FRA_j is shown with a solid line, and the skeletal information based on the second comparison frame FRB_j is shown with a dotted line. Note that the display mode for distinguishing the skeletal information is not limited to solid and dotted lines and may be changed as appropriate.

[0145] Furthermore, the display item P3 relating to the comparative frame group FG_j is displayed in the same display area as the display item P5.

[0146] Referring again to Figure 14, for display item P2, the comparison screen SC shows an example in which it includes both the frame positions of the first comparison frames FRA_1 to FRA_N in the first video and the frame positions of the second comparison frames FRB_1 to FRB_N in the second video.

[0147] 14, the frame positions of the first comparison frames FRA_4 to FRA_N-1 and the second comparison frames FRB_4 to FRB_N-1 are not shown. Furthermore, the display item P2 may be displayed with a line or the like connecting the frame positions of one or more corresponding first comparison frames FRA_j and second comparison frames FRB_j to associate them. Here, j is an integer between 1 and N.

[0148] 14 shows an example in which the display item P4 displays the similarity between the actions shown in the first video and the second video, thereby showing how similar the overall actions shown in the first video are to the overall actions shown in the second video.

[0149] 14 and 15 are examples of the configuration of the comparison screen SC, and the configuration of the comparison screen SC may be changed as appropriate.

[0150] (Actions and Effects) As described above, according to this embodiment, the information processing device 103 includes a first selection unit 132 and a second selection unit 134. The first selection unit 132 selects first to Nth first comparison frames from the first video, where N is an integer equal to or greater than 2. The second selection unit 134 selects first to Nth second comparison frames from the second video, which correspond to the first to Nth first comparison frames, respectively. The order of the first to Nth first comparison frames follows the chronological order in the first video. The order of the first to Nth second comparison frames follows the chronological order in the second video.

[0151] According to this, the first through Nth second comparison frames are selected to correspond to each of the multiple first comparison frames so that their chronological order in the second video matches the chronological order of the first through Nth first comparison frames in the first video. This makes it possible to compare the contents of the first and second videos using frames suitable for comparison. This makes it easy to compare the contents of different videos.

[0152] According to this embodiment, when selecting the ith second comparison frame, where i is an integer between 2 and N, a first similarity between the ith first comparison frame and each second frame that is chronologically later than the (i-1)th second comparison frame in the second video is used.

[0153] This allows the first through Nth second comparison frames to be selected that are similar to the first through Nth first comparison frames, respectively. Therefore, the contents of the first and second videos can be compared using frames suitable for comparison. This makes it possible to easily compare the contents of different videos.

[0154] According to this embodiment, the i-th second comparative frame is selected based on the first similarity from among second frames that are chronologically later than the (i-1)-th second comparative frame in the second video.

[0155] This allows the first through Nth second comparison frames to be selected so that their order in the second video matches the order of the first through Nth first comparison frames in the first video. This allows the content of the first and second videos to be compared using frames suitable for comparison. This makes it easy to compare the content of different videos.

[0156] According to this embodiment, the i-th second comparison frame is the frame with the largest first similarity among the second frames that are chronologically later than the (i-1)-th second comparison frame in the second video.

[0157] This increases the likelihood of selecting the first through Nth second comparison frames that are most similar to the first through Nth first comparison frames, respectively. Therefore, the content of the first and second videos can be compared using frames suitable for comparison. Therefore, the content of different videos can be easily compared.

[0158] According to this embodiment, the i-th second comparison frame is a second frame that is chronologically later than the (i-1)-th second comparison frame in the second video, and has a first similarity that is equal to or greater than a predetermined first threshold and is the frame with the largest first similarity.

[0159] This reduces the possibility of comparing the contents of the first and second videos using first and second comparison frames that are not very similar, and allows the contents of the first and second videos to be compared using frames that are suitable for comparison, thereby making it possible to accurately and easily compare the contents of different videos.

[0160] According to this embodiment, if the first similarity is less than the first threshold for all second frames that are chronologically later than the (i-1)th second comparison frame in the second video, the i-th second comparison frame is not selected.

[0161] This reduces the possibility of comparing the contents of the first and second videos using first and second comparison frames that are not very similar, and allows the contents of the first and second videos to be compared using frames that are suitable for comparison, thereby making it possible to accurately and easily compare the contents of different videos.

[0162] According to this embodiment, the second frame that is chronologically later than the i-1th second comparison frame in the second video is a predetermined number of second frames that are chronologically later than the i-1th second comparison frame in the second video.

[0163] This allows the number of second frames referenced to calculate the first similarity to be reduced to a predetermined number, thereby reducing the processing load for comparing the contents of different videos.

[0164] According to this embodiment, the i-th first comparison frame is the first frame that is closest to the i-1th first comparison frame among the first frames that are chronologically later than the i-1th first comparison frame in the first video, and whose similarity to the i-1th first comparison frame is equal to or less than a third threshold.

[0165] This allows a first frame that is included near the (i-1)th first comparison frame in the first video and that is not very similar to the (i-1)th first comparison frame to be selected as the i-th first comparison frame, thereby making it possible to select a first frame that is suitable for comparing different videos as the i-th first comparison frame.

[0166] According to this embodiment, the third threshold is adjusted so that the number of first comparative frames selected from the first video becomes a predetermined reference number.

[0167] This allows an appropriate number of first comparison frames to be selected from the first video, thereby making it possible to compare the contents of different videos while reducing the processing load required for comparing the contents of different videos.

[0168] According to this embodiment, the information processing device 103 includes a display control unit 136 that displays a comparison screen on the display unit 123. The comparison screen includes, for each of at least one comparison frame group consisting of corresponding first comparison frames and second comparison frames, an area that associates the corresponding first comparison frame and second comparison frame.

[0169] This allows a user viewing the comparison screen to easily recognize the differences between the first comparison frame and the second comparison frame, making it possible to easily compare the contents of different videos.

[0170] According to this embodiment, the comparison screen further includes at least one of the positions of the first comparison frames in the first video and the positions of the second comparison frames in the second video.

[0171] This allows a user viewing the comparison screen to easily recognize the time points in the first and second videos at which the first and second comparison frames are being compared, and to easily recognize differences in the timing of actions, etc. This makes it possible to easily compare the contents of different videos.

[0172] According to this embodiment, the comparison screen further includes the similarity between at least the first comparison frame and the second comparison frame included in the comparison frame group.

[0173] This allows a user viewing the comparison screen to objectively recognize the differences between the first and second comparison frames, making it possible to objectively and easily compare the content of different videos.

[0174] According to this embodiment, the comparison screen further includes a degree of similarity in action between the first video and the second video.

[0175] This allows the overall differences between the first and second images to be objectively recognized, thereby making it possible to objectively and easily compare the contents of the different images.

[0176] According to this embodiment, the comparison screen further includes information associating skeletal information of the object shown in each of the corresponding first and second comparison frames.

[0177] This allows a user viewing the comparison screen to clearly recognize the differences between the first and second comparison frames, making it possible to easily compare the contents of different videos.

[0178] According to this embodiment, there are a plurality of comparative frame groups. The plurality of comparative frame groups are arranged on the comparison screen in such a manner that the first comparative frames included in each comparative frame group are arranged in chronological order in the first video.

[0179] This allows a user viewing the comparison screen to easily recognize the differences between the first and second comparison frames according to the order in the first and second videos, making it possible to easily compare the content of the different videos.

[0180] [Embodiment 2] In embodiment 2, another example of a method for selecting a second comparison frame will be described. Except for the method for selecting a second comparison frame, the information processing system 100 and information processing in this embodiment may be configured in substantially the same manner as those described in embodiment 1.

[0181] In other examples of the method for selecting the second comparative frame, the method for selecting the second comparative frame FRB_1 may be the same as the example of the method for selecting the second comparative frame described in the first embodiment.

[0182] Furthermore, when selecting the second comparison frame FRB_i, the second selection unit 134 may also select from second frames in the second video that are later than the second comparison frame FRB_i-1 in other examples of the method for selecting the second comparison frame. Therefore, also in other examples of the method for selecting the second comparison frame, the order of the second comparison frames FRB_1 to FRB_N in the second video matches the order of the first comparison frames FRA_1 to FRA_N in the first video, as described in the first embodiment.

[0183] The difference between the example of the method for selecting a second comparison frame described in embodiment 1 and the other examples of the method for selecting a second comparison frame described in this embodiment is generally in the second selection process for selecting the second comparison frame FRB_i corresponding to the first comparison frame FRA_i, as follows:

[0184] In the first embodiment, an example has been described in which the second frames of the selection candidates are all or a predetermined number of second frames that are after the second comparison frame FRB_i-1 in the second video. Then, an example has been described in which the second frame with the largest first similarity among the second frames of the selection candidates is selected as the second comparison frame FRB_i.

[0185] In this embodiment, for example, multiple candidate frames to be candidates for the second comparison frame FRB_i are selected from second frames that are later than the second comparison frame FRB_i-1 in the second video based on the first similarity.

[0186] Then, the second comparison frame FRB_i is selected, for example, from among a plurality of candidate frames based on the similarity between each of the plurality of candidate frames and each of a plurality of first frames included in the first video (this similarity will hereinafter also be referred to as "second similarity").

[0187] In more detail, for example, for each of the plurality of candidate frames, a first similar frame that is the most similar first frame is identified based on the second similarity to each of the first frames. Among the plurality of first similar frames, a candidate frame that corresponds to a first similar frame within a predetermined range from the first comparison frame FRA_i in the first video and has the largest first similarity or second similarity is selected as the second comparison frame FRB_i.

[0188] Hereinafter, details of another example of the second selection process for selecting the second comparative frame FRB_i corresponding to the first comparative frame FRA_i will be described with reference to the drawings.

[0189] FIG. 16 is a diagram showing another example of the second selection process for selecting the second comparative frame FRB_i corresponding to the first comparative frame FRA_i in step S104.

[0190] The similarity obtaining unit 135 obtains the similarity (first similarity) between the first comparative frame FRA_i and each of the second frames that come after the second comparative frame FRB_i-1 in the second video (step S104g).

[0191] The second selection unit 134 uses the first similarity calculated in step S104g to select candidate frames from the second frames (step S104h).

[0192] The candidate frame is a second frame that is a candidate for the second comparison frame FRB_i. For example, the second selection unit 134 selects a second frame whose first similarity calculated in step S104g is equal to or greater than a predetermined candidate threshold as a candidate frame that is a candidate for the second comparison frame FRB_i.

[0193] Here, if there is one second frame whose first similarity is equal to or greater than the candidate threshold, the second selection unit 134 may select that second frame as the second comparison frame FRB_i. Therefore, hereinafter, an example will be described in which there are multiple second frames whose first similarity is equal to or greater than the candidate threshold, and multiple candidate frames are selected.

[0194] The similarity obtaining unit 135 obtains the similarity (second similarity) between each of the candidate frames selected in step S104h and each of the plurality of first frames included in the first video (step S104i).

[0195] In step S104i, the plurality of first frames for which similarity to each of the candidate frames is calculated may be, for example, all first frames included in the first video. Alternatively, the plurality of first frames may be, for example, first frames within a predetermined range from the first comparison frame FRA_i in the first video. This predetermined range may be determined in advance, for example, by the number of frames in chronological order in the first video, the time in the first video, or the like.

[0196] The second selection unit 134 determines whether or not any of the second similarities calculated in step S104i is equal to or greater than a predetermined second threshold value (step S104j).

[0197] If there is no second similarity greater than or equal to the second threshold (step S104j; No), the second selection unit 134 does not select the second comparison frame FRB_i and ends the second selection process. Therefore, if the second similarity for all candidate frames is less than the second threshold, the second comparison frame FRB_i is not selected.

[0198] If there is a second similarity greater than or equal to the second threshold (step S104j; Yes), the second selection unit 134 identifies a first similar frame based on the second similarity for each candidate frame whose second similarity is greater than or equal to the second threshold (step S104k).

[0199] As described above, the first similar frame is the first frame that is most similar to each of the candidate frames, for example, the first similar frame is the first frame that has the largest second similarity to the candidate frame.

[0200] The second selection unit 134 selects, as the second comparison frame FRB_i, a candidate frame that corresponds to a first similar frame within a predetermined range from the first comparison frame FRA_i in the first video and has the largest first similarity calculated in step S104g or the largest second similarity calculated in step S104i (step S104l).This completes the second selection process.

[0201] That is, the second comparison frame FRB_i is one of the candidate frames corresponding to the first similar frames within a predetermined range from the first comparison frame FRA_i in the first video among the plurality of first similar frames identified in step S104k. Furthermore, among such candidate frames, the candidate frame having the greatest first similarity or second similarity is selected as the second comparison frame FRB_i. This predetermined range may be determined in advance, for example, by the number of chronologically ordered frames in the first video or the time in the first video.

[0202] This predetermined range may be the same as the predetermined range used in step S104i, or may be different.

[0203] In more detail, for example, the second selection unit 134 may determine whether the first similar frame is within a predetermined range from the first comparison frame FRA_i in the first video, starting with the first similar frame having the greatest first similarity or the second similarity. Then, the second selection unit 134 may identify the first similar frame that is determined to be within the predetermined range from the first comparison frame FRA_i, and select a candidate frame corresponding to the identified first similar frame as the second comparison frame FRB_i.

[0204] (Actions and Effects) As described above, according to this embodiment, when selecting the ith second comparison frame, a plurality of candidate frames are selected as candidates for the ith second comparison frame from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video, based on the first similarity. The ith second comparison frame is selected from the plurality of candidate frames based on the second similarity between each of the plurality of candidate frames and each of the plurality of first frames included in the first video.

[0205] This allows the first through Nth second comparison frames to be selected that are similar to the first through Nth first comparison frames, respectively. Therefore, the contents of the first and second videos can be compared using frames suitable for comparison. This makes it possible to easily compare the contents of different videos.

[0206] According to this embodiment, the plurality of candidate frames are second frames that are chronologically subsequent to the (i-1)th second comparison frame in the second video and have a first similarity equal to or greater than a predetermined candidate threshold. The i-th second comparison frame is a candidate frame that corresponds to a first similar frame within a predetermined range from the i-th first comparison frame in the first video and has the greatest first similarity or the greatest second similarity among the plurality of first similar frames that are first frames having the greatest second similarity with each of the plurality of candidate frames.

[0207] This increases the likelihood of selecting the first through Nth second comparison frames that are most similar to the first through Nth first comparison frames, respectively. Therefore, the content of the first and second videos can be compared using frames suitable for comparison. This makes it possible to easily compare the content of different videos.

[0208] According to this embodiment, the i-th second comparison frame corresponds to a first similar frame within a predetermined range, has a second similarity greater than or equal to a predetermined second threshold, and is a candidate frame with the largest first similarity or second similarity.

[0209] This reduces the possibility of comparing the contents of the first and second videos using first and second comparison frames that are not very similar, and allows the contents of the first and second videos to be compared using frames that are suitable for comparison, thereby making it possible to accurately and easily compare the contents of different videos.

[0210] Although the embodiments and modifications of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0211] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps performed in each embodiment is not limited to the order shown. In each embodiment, the order of steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments and variations can be combined as long as the content is not contradictory.

[0212] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0213] 1. An information processing device comprising: first selection means for selecting first to Nth first comparison frames from a first video, where N is an integer of 2 or greater; and second selection means for selecting first to Nth second comparison frames from a second video corresponding to the first to Nth first comparison frames, respectively, wherein the first to Nth first comparison frames are ordered in chronological order in the first video, and the first to Nth second comparison frames are ordered in chronological order in the second video, and each of the first to Nth second comparison frames is a plurality of frames that show the same operating state as the corresponding first to Nth first comparison frames. 2. The information processing device described in 1., wherein i is an integer between 2 and N, and wherein when selecting the ith second comparison frame, a first similarity between the ith first comparison frame and each of the second frames that comes chronologically after the (i-1)th second comparison frame in the second video is used. 3. The information processing device described in 2., wherein the i-th second comparison frame is selected based on the first similarity from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 4. The information processing device described in 3., wherein the i-th second comparison frame is a frame with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 5. The information processing device described in 4., wherein the i-th second comparison frame is a frame with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video, the first similarity being equal to or greater than a predetermined first threshold. 6. The information processing device described in 5., wherein the i-th second comparison frame is not selected if the first similarity is less than the first threshold for all second frames that are chronologically later than the (i-1)th second comparison frame in the second video.7. The information processing device described in any one of 2. to 6., wherein the second frames that are chronologically later than the i-1th second comparison frame in the second video are a predetermined number of second frames that are chronologically later than the i-1th second comparison frame in the second video. 8. The information processing device described in 2., wherein when selecting the i-th second comparison frame, a plurality of candidate frames are selected as candidates for the i-th second comparison frame from among second frames that are chronologically later than the i-1th second comparison frame in the second video based on the first similarity, and the i-th second comparison frame is selected from the plurality of candidate frames based on the second similarity between each of the plurality of candidate frames and each of the plurality of first frames included in the first video. 9. The information processing device described in 8., wherein the plurality of candidate frames are second frames that are second frames that are later in chronological order than the (i-1)th second comparison frame in the second video, and for which the first similarity is equal to or greater than a predetermined candidate threshold, and the ith second comparison frame is a candidate frame that corresponds to a first similar frame within a predetermined range in chronological order from the ith first comparison frame in the first video and has the largest first similarity, among a plurality of first similar frames that are first frames that have the largest second similarity with each of the plurality of candidate frames. 10. The information processing device described in 9., wherein the ith second comparison frame is a candidate frame that corresponds to a first similar frame within the predetermined range, has the second similarity equal to or greater than a predetermined second threshold, and has the largest second similarity. The information processing device described in any one of 2. to 10., wherein the i-th first comparison frame is a first frame that is closest to the i-1th first comparison frame among first frames that are chronologically later than the i-1th first comparison frame in the first video, and whose similarity to the i-1th first comparison frame is equal to or less than a third threshold. 12. The information processing device described in 11., wherein the third threshold is adjusted so that the number of first comparison frames selected from the first video becomes a predetermined reference number.13. The information processing device according to any one of 1. to 12., further comprising display control means for causing a display means to display a comparison screen in which information related to the comparison frames is arranged in chronological order. 14. The information processing device according to any one of 1. to 13., further comprising display control means for causing a display means to display a comparison screen in which the corresponding first comparison frames and second comparison frames are associated with each other, for each of at least one comparison frame group made up of the corresponding first comparison frames and second comparison frames. 15. The information processing device according to 14., wherein the comparison screen further includes at least one of the frame positions of each of the first comparison frames in the first video and the frame positions of each of the second comparison frames in the second video. 16. The information processing device according to 14. or 15., wherein the comparison screen further includes a similarity between the first comparison frame and the second comparison frame included in at least the comparison frame group. 17. The comparison screen further includes an action similarity that is a similarity between actions shown in the first video and the second video. 18. The information processing device described in any one of 14. to 16.. 18. The information processing device described in any one of 14. to 17., wherein the comparison screen further includes information associating skeletal information of the object shown in each of the corresponding first comparison frame and second comparison frame. 19. The information processing device described in any one of 14. to 18., wherein there are a plurality of comparison frame groups, and the plurality of comparison frame groups are arranged on the comparison screen such that the first comparison frames included in each comparison frame group are arranged in chronological order in the first video. 20. An information processing system comprising: the information processing device described in any one of 1. to 19.; a first video storage device that stores the first video; and a terminal device that captures the second video.21. An information processing method, comprising one or more computers selecting, from a first video, first to N-th first comparison frames, where N is an integer equal to or greater than 2, and selecting, from a second video, first to N-th second comparison frames corresponding to the first to N-th first comparison frames, respectively, wherein the first to N-th first comparison frames are ordered in chronological order in the first video, the first to N-th second comparison frames are ordered in chronological order in the second video, and each of the first to N-th second comparison frames is a plurality of frames that show the same motion state as the corresponding first to N-th first comparison frames. 22. The information processing method described in 21., wherein i is an integer equal to or greater than 2 and equal to N, and wherein, when selecting the ith second comparison frame, a first similarity between the ith first comparison frame and each of second frames that chronologically follows the (i-1)th second comparison frame in the second video is used. 23. The information processing method described in 22., wherein the i-th second comparison frame is selected based on the first similarity from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 24. The information processing method described in 23., wherein the i-th second comparison frame is the frame with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 25. The information processing method described in 24., wherein the i-th second comparison frame is the frame with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video, the first similarity being equal to or greater than a predetermined first threshold. 26. The information processing method described in 25., wherein the i-th second comparison frame is not selected if the first similarity is less than the first threshold for all second frames that are chronologically later than the (i-1)th second comparison frame in the second video.27. The information processing method described in any one of 22. to 26., wherein the second frames that are chronologically later than the (i-1)th second comparison frame in the second video are a predetermined number of second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 28. The information processing method described in 22., wherein when selecting the i-th second comparison frame, a plurality of candidate frames are selected as candidates for the i-th second comparison frame from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video based on the first similarity, and the i-th second comparison frame is selected from the plurality of candidate frames based on the second similarity between each of the plurality of candidate frames and each of the plurality of first frames included in the first video. 29. The information processing method described in 28., wherein the plurality of candidate frames are second frames that are second frames that are later in chronological order than the (i-1)th second comparison frame in the second video, and for which the first similarity is equal to or greater than a predetermined candidate threshold, and the ith second comparison frame is a candidate frame that corresponds to a first similar frame within a predetermined range in chronological order from the ith first comparison frame in the first video and has the largest first similarity, among a plurality of first similar frames that are first frames having the largest second similarity with each of the plurality of candidate frames. 30. The information processing method described in 29., wherein the ith second comparison frame is a candidate frame that corresponds to a first similar frame within the predetermined range, has the second similarity equal to or greater than a predetermined second threshold, and has the largest second similarity. 31. The information processing method described in any one of 22. to 30., wherein the i-th first comparison frame is a first frame that is closest to the i-1th first comparison frame among first frames that are chronologically later than the i-1th first comparison frame in the first video, and whose similarity to the i-1th first comparison frame is equal to or less than a third threshold. 32. The information processing method described in 31., wherein the third threshold is adjusted so that the number of first comparison frames selected from the first video becomes a predetermined reference number.33. The information processing method according to any one of 21. to 32., further comprising causing a display unit to display a comparison screen in which information related to the comparison frames is arranged in chronological order. 34. The information processing method according to any one of 21. to 33., further comprising causing a display unit to display a comparison screen in which the corresponding first comparison frame and second comparison frame are associated with each other, for each of at least one comparison frame group consisting of the corresponding first comparison frame and second comparison frame. 35. The information processing method according to 34., wherein the comparison screen further includes at least one of the frame positions of each of the first comparison frames in the first video and the frame positions of each of the second comparison frames in the second video. 36. The information processing method according to 34. or 35., wherein the comparison screen further includes a similarity between the first comparison frame and the second comparison frame included in at least the comparison frame group. 37. The comparison screen further includes an action similarity, which is a similarity between actions shown in the first video and the second video. The information processing method described in any one of 34. to 36. 38. The information processing method described in any one of 34. to 37., wherein the comparison screen further includes information associating skeletal information of the object shown in each of the corresponding first comparison frame and second comparison frame. 39. The information processing method described in any one of 34. to 38., wherein there are a plurality of comparison frame groups, and the first comparison frames included in each of the plurality of comparison frame groups are arranged on the comparison screen in chronological order in the first video.40. A program for causing one or more computers to: select first through Nth first comparison frames from a first video, where N is an integer equal to or greater than 2; and select first through Nth second comparison frames from a second video that correspond to the first through Nth first comparison frames, respectively, such that the first through Nth first comparison frames are ordered in chronological order in the first video, the first through Nth second comparison frames are ordered in chronological order in the second video, and each of the first through Nth second comparison frames is a plurality of frames that show the same motion state as the corresponding first through Nth first comparison frames. 41. The program described in 39., where i is an integer equal to or greater than 2 and equal to N, and in which, when selecting the ith second comparison frame, a first similarity between the ith first comparison frame and each of the second frames that chronologically follows the (i-1)th second comparison frame in the second video is used. 42. The program described in 41., wherein the i-th second comparison frame is selected based on the first similarity from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 43. The program described in 42., wherein the i-th second comparison frame is the frame with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 44. The program described in 43., wherein the i-th second comparison frame is the frame with the largest first similarity and with the largest first similarity among second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 45. The program described in 44., wherein the i-th second comparison frame is not selected if the first similarity is less than the first threshold for all second frames that are chronologically later than the (i-1)th second comparison frame in the second video.46. ​​The program described in any one of 41. to 45., wherein the second frame that is chronologically later than the (i-1)th second comparison frame in the second video is a predetermined number of second frames that are chronologically later than the (i-1)th second comparison frame in the second video. 47. The program described in 41., wherein when selecting the i-th second comparison frame, a plurality of candidate frames are selected as candidates for the i-th second comparison frame from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video based on the first similarity, and the i-th second comparison frame is selected from the plurality of candidate frames based on the second similarity between each of the plurality of candidate frames and each of the plurality of first frames included in the first video. 48. The program described in 47., wherein the plurality of candidate frames are second frames that are chronologically subsequent to the (i-1)th second comparison frame in the second video, and have the first similarity equal to or greater than a predetermined candidate threshold, and the ith second comparison frame is a candidate frame that corresponds to a first similar frame within a predetermined range in chronological order from the ith first comparison frame in the first video and has the highest first similarity, among a plurality of first similar frames that are first frames having the highest second similarity with each of the plurality of candidate frames. 49. The program described in 48., wherein the ith second comparison frame is a candidate frame that corresponds to a first similar frame within the predetermined range, has the second similarity equal to or greater than a predetermined second threshold, and has the highest second similarity. 50. The program described in any one of 41. to 49., wherein the i-th first comparison frame is a first frame that is closest to the i-1th first comparison frame among first frames that are chronologically later than the i-1th first comparison frame in the first video, and whose similarity to the i-1th first comparison frame is equal to or less than a third threshold. 51. The program described in 50., wherein the third threshold is adjusted so that the number of first comparison frames selected from the first video becomes a predetermined reference number.52. The program described in any one of 40. to 51., further causing a display unit to display a comparison screen in which information related to the comparison frames is arranged in chronological order. 53. The program described in any one of 40. to 52., further causing a display unit to display a comparison screen in which the corresponding first comparison frame and second comparison frame are associated with each other, for at least one comparison frame group made up of the corresponding first comparison frame and second comparison frame. 54. The program described in 53., wherein the comparison screen further includes at least one of the frame positions of the first comparison frames in the first video and the frame positions of the second comparison frames in the second video. 55. The program described in 53. or 54., wherein the comparison screen further includes a similarity between the first comparison frame and the second comparison frame included in at least the comparison frame group. 56. The comparison screen further includes an action similarity, which is a similarity between actions shown in the first video and the second video. 57. The program described in any one of 53. to 56., wherein the comparison screen further includes information associating skeletal information of the object shown in each of the corresponding first comparison frame and second comparison frame. 58. The program described in any one of 53. to 57., wherein there are a plurality of comparison frame groups, and the plurality of comparison frame groups are arranged on the comparison screen such that the first comparison frames included in each comparison frame group are arranged in chronological order in the first video.

[0214] This application claims priority based on Japanese Patent Application No. 2023-146749, filed September 11, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0215] REFERENCE SIGNS LIST 100 Information processing system 101 Video storage device 102 Terminal device 103 Information processing device 111 Video storage unit 113 Video communication control unit 121 Input unit 122 Shooting unit 123 Display unit 124 Terminal communication control unit 131 First video acquisition unit 132 First selection unit 133 Second video acquisition unit 134 Second selection unit 135 Similarity acquisition unit 136 Display control unit

Claims

1. An information processing device comprising: first selection means for selecting 1st to Nth first comparison frames from a first video, where N is an integer equal to or greater than 2; and second selection means for selecting 1st to Nth second comparison frames from a second video corresponding to the 1st to Nth first comparison frames, respectively; the order of the 1st to Nth first comparison frames follows chronological order in the first video; the order of the 1st to Nth second comparison frames follows chronological order in the second video; and each of the 1st to Nth second comparison frames is a plurality of frames that show the same operating state as the 1st to Nth corresponding first comparison frames.

2. The information processing device of claim 1, wherein i is an integer between 2 and N, and when selecting the i-th second comparison frame, a first similarity between the i-th first comparison frame and each of the second frames that are chronologically later than the (i-1)th second comparison frame in the second video is used.

3. An information processing device as described in claim 2, wherein the i-th second comparison frame is selected based on the first similarity from among second frames that are chronologically later than the i-1-th second comparison frame in the second video.

4. An information processing device as described in claim 3, wherein the i-th second comparison frame is a frame having the largest first similarity among second frames that are chronologically later than the i-1-th second comparison frame in the second video.

5. The information processing device of claim 2, wherein when selecting the ith second comparison frame, a plurality of candidate frames are selected as candidates for the ith second comparison frame from among second frames that are chronologically later than the (i-1)th second comparison frame in the second video based on the first similarity, and the ith second comparison frame is selected from among the plurality of candidate frames based on a second similarity between each of the plurality of candidate frames and each of the plurality of first frames included in the first video.

6. The information processing device of claim 5, wherein the plurality of candidate frames are second frames that are chronologically subsequent to the (i-1)th second comparison frame in the second video and have the first similarity equal to or greater than a predetermined candidate threshold, and the ith second comparison frame is a candidate frame that corresponds to the first similar frame within a predetermined range from the ith first comparison frame in the first video and has the highest first similarity among a plurality of first similar frames that are first frames having the highest second similarity between each of the plurality of candidate frames.

7. An information processing device as claimed in any one of claims 2 to 6, wherein the i-th first comparison frame is a first frame that is chronologically later than the i-1-th first comparison frame in the first video, is closest to the i-1-th first comparison frame, and has a similarity to the i-1-th first comparison frame that is equal to or less than a third threshold.

8. The information processing apparatus according to any one of claims 1 to 7, further comprising display control means for causing a display means to display a comparison screen in which information related to the comparison frames is arranged in chronological order.

9. An information processing method including one or more computers selecting, from a first video, 1st to Nth first comparison frames, where N is an integer equal to or greater than 2, and selecting, from a second video, 1st to Nth second comparison frames corresponding to the 1st to Nth first comparison frames, respectively, wherein the order of the 1st to Nth first comparison frames follows the chronological order in the first video, the order of the 1st to Nth second comparison frames follows the chronological order in the second video, and each of the 1st to Nth second comparison frames is a plurality of frames that show the same operating state as the 1st to Nth corresponding first comparison frames.

10. A recording medium having recorded thereon a program for causing one or more computers to: select from a first video 1st through Nth first comparison frames, where N is an integer equal to or greater than 2; and select from a second video 1st through Nth second comparison frames corresponding to the first through Nth first comparison frames, respectively; such that the order of the first through Nth first comparison frames follows the chronological order in the first video; the order of the first through Nth second comparison frames follows the chronological order in the second video; and each of the first through Nth second comparison frames is a plurality of frames that show the same operating state as the first through Nth corresponding first comparison frames.

Images