Information processing device, information processing method, and program

Abstract

To suitably compare a plurality of moving images.SOLUTION: An information processing device (2) comprises: a display unit (123) which displays operators for selection (M_BT4, FL_BT3) for selecting one of a plurality of reproduction start positions and an operator for reproduction start (P_BT2) for simultaneously starting reproduction of a plurality of moving images; a reception unit (124) which receives an operation input; and a processing execution unit (125) which when the operator for reproduction start is operated, simultaneously starts reproduction of the plurality of moving images from the reproduction start position selected by the operators for selection.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A technique is known in which a predetermined operation is executed on a subject in order to evaluate the walking ability of the subject, and an evaluation is performed based on the operation.

[0003] Non-Patent Document 1 discloses a Timed Up & Go Test (TUG test) for evaluating movements from the state of sitting on a chair, standing up, walking 3 m, making a U-turn, and sitting down again.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a subject executes a predetermined operation multiple times, a technique for comparing the predetermined operations executed multiple times is required. For example, when a subject executes the TUG test multiple times, a technique for comparing how the subject's walking has changed is required.

[0006] <​​​​​​​To solve the above problems, an information processing device according to one aspect of the present invention includes a display unit that displays a display screen including a display area for displaying each of a plurality of moving images, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the plurality of moving images, and a playback start operator for simultaneously starting playback of the plurality of moving images, a reception unit that receives operation input via the selection operator and the playback start operator, and a processing execution unit that, when the playback start operator is operated, simultaneously starts playback of the plurality of moving images from the playback start position selected by the selection operator. [Effects of the Invention]

[0008] According to one aspect of the present invention, multiple moving images can be more preferably compared. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the configuration of an information processing system according to Embodiment 1 of the present invention. [Figure 2] This figure shows an overview of the information processing system according to Embodiment 1 of the present invention. [Figure 3] This figure shows an example of skeletal information in Embodiment 1 of the present invention. [Figure 4] This diagram shows the actions of participants during a TUG test. [Figure 5] This is an example of a graph showing the relationship between the subject's body width and elapsed time in Embodiment 1 of the present invention. [Figure 6] This figure shows an example of the travel distance calculated by the specific unit in Embodiment 1 of the present invention. [Figure 7] This is an example of a graph showing the relationship between the distance traveled by the subject's feet and the elapsed time in Embodiment 1 of the present invention. [Figure 8] This is an example of a graph showing the relationship between the length of the subject's thigh and the elapsed time in Embodiment 1 of the present invention. [Figure 9] This figure shows an example of the process by which the specific unit generates interval information in Embodiment 1 of the present invention. [Figure 10] It is a sequence diagram showing the processing flow in the information processing system according to Embodiment 1 of the present invention. [Figure 11] It is a diagram showing an example of a display screen displayed on a display in Embodiment 1 of the present invention. [Figure 12] It is a diagram showing the configuration of the information processing system according to Embodiment 2 of the present invention. [Figure 13] It is a diagram showing an example of a display screen displayed by a display unit in Embodiment 2 of the present invention. [Figure 14] It is a sequence diagram showing the processing flow in the information processing system according to Embodiment 2 of the present invention. [Figure 15] It is a block diagram showing the configuration of the information processing apparatus according to Embodiment 3 of the present invention. [Figure 16] It is a block diagram exemplifying the physical configuration of a computer used as the information processing apparatus according to Embodiment 1 and Embodiment 2 of the present invention. [Figure 17] It is a block diagram exemplifying the physical configuration of a computer used as the information processing apparatus according to Embodiment 3 of the present invention.

Embodiments for Carrying Out the Invention

[0010] 〔Embodiment 1〕 Hereinafter, an embodiment of the present invention will be described in detail.

[0011] (Outline of Information Processing System 100) First, the outline of the information processing system 100 according to this embodiment will be described with reference to FIG. 2. FIG. 2 is a diagram showing the outline of the information processing system 100 according to this embodiment. As shown in FIG. 2, the information processing system 100 includes an information processing apparatus 1 and an information processing apparatus 2.

[0012] In the information processing system 100, as shown in FIG. 2, the information processing device 1 and the information processing device 2 are communicably connected to each other via the network N. In the information processing system 100, the information processing device 1 acquires a moving image including a subject (hereinafter also referred to as "target person") within the angle of view. The information processing device 1 outputs the acquired moving image to the information processing device 2 via the network N. When the information processing device 2 acquires the moving image, it specifies a turning section in which the state of the target person turning is imaged among the sections included in the moving image. Then, the information processing device 2 evaluates the ability of the target person to move their own body.

[0013] The user of the information processing device 1 is not particularly limited, but examples include caregivers, physical therapists (PTs), and occupational therapists (OTs). Also, the target person is not particularly limited, but examples include care recipients and the elderly.

[0014] Note that the moving image including the target person acquired by the information processing device 1 may be a moving image including a target person performing a predetermined operation within the angle of view. Here, the predetermined operation is not particularly limited, but an example is the operation in the Timed up & go test (TUG test). Details of the TUG test will be described later.

[0015] Examples of the information processing device 1 include mobile terminals such as smartphones and tablets equipped with an imaging device (e.g., a camera) for imaging a moving image, but are not limited thereto. For example, the information processing device 1 may not be equipped with an imaging device. In this case, the information processing device 1 may be configured to acquire a moving image from another device (e.g., a camera, etc.). An example of the information processing device 2 is a server, but is not limited thereto.

[0016] The specific configuration of network N is not limited to this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public telephone network, a mobile data communication network, or a combination of these networks can be used.

[0017] (Configuration of Information Processing Device 1) Next, the configuration of the information processing device 1 will be explained using Figure 1. Figure 1 is a diagram showing the configuration of the information processing system 100 according to this embodiment.

[0018] As shown in Figure 1, the information processing device 1 includes a communication unit 10, a storage unit 11, a control unit 12, an imaging unit 13, and a display 14.

[0019] The communication unit 10 is a communication module for communicating with the information processing device 2 via the network N. The communication unit 10 supplies data acquired from the information processing device 2 via the network N to the control unit 12, and outputs data acquired from the control unit 12 to the information processing device 2 via the network N.

[0020] The memory unit 11 stores data that the control unit 12 references. Examples of data stored in the memory unit 11 include, but are not limited to, video footage and evaluation results described later.

[0021] The imaging unit 13 is a camera capable of capturing moving images. For example, the imaging unit 13 is oriented towards a subject by the user of the information processing device 1 so that the subject is included in the imaging unit 13's field of view. The imaging unit 13 then captures a moving image that includes the subject in its field of view and supplies the moving image to the control unit 12.

[0022] As shown in Figure 1, the display 14 comprises a display panel 141 and a touchpad 142.

[0023] The display panel 141 is a device that displays images supplied from the control unit 12.

[0024] The touchpad 142 is a user interface that accepts user input. The touchpad 142 is positioned so as to overlap the display panel 141, detects contact with the display panel 141, and generates position information indicating the location (coordinates) where the contact was detected. The touchpad 142 then supplies this position information to the control unit 12.

[0025] In other words, the display 14 displays the image supplied from the control unit 12 on the display panel 141, and also supplies position information generated by the touchpad 142 to the control unit 12.

[0026] Here, the information processing device 1 may be configured to have a user interface such as a keyboard, mouse, and buttons instead of the touchpad 142, and to accept operations from the user through this user interface.

[0027] (Functions of the control unit 12) The control unit 12 controls each component of the information processing device 1. As shown in Figure 1, the control unit 12 also functions as an acquisition unit 121, an output unit 122, a display unit 123, a reception unit 124, and a processing execution unit 125.

[0028] The acquisition unit 121 acquires data supplied from the communication unit 10 and the imaging unit 13. For example, the acquisition unit 121 acquires a video image from the imaging unit 13 that includes the subject in the field of view, or acquires evaluation results (described later) from the communication unit 10. The acquisition unit 121 stores the acquired data in the storage unit 11.

[0029] The output unit 122 outputs data to the information processing device 2 via the communication unit 10. For example, the output unit 122 outputs a video image stored in the storage unit 11 to the information processing device 2 via the communication unit 10.

[0030] The display unit 123 supplies the image to be displayed to the display 14 and causes the image to be displayed on the display 14. For example, the display unit 123 displays a moving image stored in the storage unit 11 on the display 14.

[0031] The reception unit 124 receives operation input from the user. For example, the reception unit 124 refers to location information supplied from the display 14 and stores operation input information indicating the user operation input indicated by the location information in the storage unit 11. The process of generating operation input information by referring to the location information acquired by the reception unit 124 will be described later.

[0032] The processing execution unit 125 executes processing in response to user operations. For example, the processing execution unit 125 refers to the operation input information stored in the storage unit 11, and if the operation input information indicates an operation to play a video, the processing execution unit 125 plays the video. The system executes the process corresponding to the input shown.

[0033] (Configuration of Information Processing Device 2) As shown in Figure 1, the information processing device 2 includes a communication unit 20, a storage unit 21, and a control unit 22.

[0034] The communication unit 20 is a communication module for communicating with the information processing device 1 via the network N. The communication unit 20 supplies data acquired from the information processing device 1 via the network N to the control unit 22, and outputs data acquired from the control unit 22 to the information processing device 1 via the network N.

[0035] The memory unit 21 stores data that the control unit 22 references. Examples of data stored in the memory unit 21 include, but are not limited to, video footage, section information (described later), and evaluation results (described later).

[0036] (Functions of the control unit 22) The control unit 22 controls each component of the information processing device 2. As shown in Figure 1, the control unit 22 also functions as an acquisition unit 221, an extraction unit 222, an identification unit 223, an evaluation unit 224, and an output unit 225.

[0037] The acquisition unit 221 acquires data supplied from the communication unit 20. For example, the acquisition unit 221 acquires a video image from the communication unit 20 that includes the subject in its field of view. The acquisition unit 221 stores the acquired data in the storage unit 21.

[0038] The extraction unit 222 applies skeletal extraction processing to the video image and extracts skeletal information indicating the skeleton of the subject included in the video image. For example, the extraction unit 222 retrieves the video image acquired by the acquisition unit 221 from the storage unit 21 and extracts the subject's skeletal information from the video image. The extraction unit 222 stores the extracted skeletal information in the storage unit 21. The skeletal information extraction algorithm executed by the extraction unit 222 is not particularly limited, but as an example, it is preferable to use an algorithm that can detect the subject's shoulders, hips, feet, and knees.

[0039] Figure 3 shows an example of skeletal information extracted by the extraction unit 222. Figure 3 is a diagram showing an example of skeletal information in this embodiment. As shown in Figure 3, the extraction unit 222 extracts, for example, left shoulder LS, right shoulder RS, left hip LW, right hip RW, left knee LK, right knee RK, left heel LH, right heel RH, left toe LT, and right toe RT as skeletal information indicating the skeleton of the subject TP.

[0040] The identification unit 223 identifies a section within the video where the subject is performing a predetermined action. For example, the identification unit 223 obtains the video acquired by the acquisition unit 221 and the skeletal information extracted by the extraction unit 222 from the storage unit 21, and based on the skeletal information, identifies a turning section within the video in which the subject is shown turning. Alternatively, the identification unit 223 may identify at least one of the following based on the skeletal information: a walking section in which the subject is shown walking, a standing motion section in which the subject is shown standing up, and a sitting motion section in which the subject is shown sitting down. The identification unit 223 stores the section information indicating the identified section in the storage unit 21. An example of the processing performed by the identification unit 223 will be described later.

[0041] The evaluation unit 224 evaluates the movements of the subject included in the video. For example, the evaluation unit 224 obtains the skeletal information extracted by the extraction unit 222 and the section information identified by the identification unit 223 from the storage unit 21, and evaluates the following items regarding the subject's movements by referring to the skeletal information and section information. • Walking speed • Unsteadiness • Left-right difference ·rhythm • Duration ·comprehensive evaluation The evaluation unit 224 stores the evaluation results in the storage unit 21. An example of the processing performed by the evaluation unit 224 will be described later.

[0042] The output unit 225 outputs data stored in the storage unit 21 via the communication unit 20. For example, the output unit 225 outputs evaluation results stored in the storage unit 21 via the communication unit 20.

[0043] (TUG test) In this embodiment, we will describe an example in which the identification unit 223 identifies the interval in which the subject performs each action in the TUG test when the subject performs a predetermined action in the TUG test. However, this does not limit the predetermined actions of the subject in this specification. First, we will explain each action in the TUG test using Figure 4. Figure 4 is a diagram showing the actions of the subject in the TUG test.

[0044] The TUG test is a test in which a subject performs a set of predetermined actions, and each of these actions is evaluated. As shown in Figure 4, the TUG test begins with the subject sitting in a chair (hereinafter referred to as the "seated position"). First, the subject stands up from the seated position (hereinafter referred to as the "standing up action"), walks 3 meters forward (hereinafter referred to as "walk 1"), and turns around (hereinafter referred to as "turn 1"). Next, the subject walks 3 meters forward again (hereinafter referred to as "walk 2"), returns to the front of the chair, and turns around to sit in the chair (hereinafter referred to as "turn 2"). Finally, the subject sits in the chair (hereinafter referred to as the "seated action") and returns to the seated position.

[0045] (An example of processing performed by the specific unit 223) An example of the processing performed by the identification unit 223 will be explained using Figures 5 to 9. In the following, unless otherwise specified, the description will be of the case in which the identification unit 223 refers to skeletal information extracted by the extraction unit 222 from a video image taken from the front of the person undergoing the TUG test. In addition, the "predetermined time" in the following is not particularly limited, but examples include 1 second or 1 frame of the video image.

[0046] (An example of the process by which the specific unit 223 identifies the turning section) An example of the process by which the specific unit 223 identifies the turning section will be described.

[0047] The identification unit 223 identifies the rotational section by calculating the subject's body width in the video based on the skeletal information extracted by the extraction unit 222. As an example of a configuration for calculating body width, the identification unit 223 identifies the positions of both of the subject's shoulders by referring to the skeletal information and calculates the subject's shoulder width (distance between the right and left shoulders) in the video at predetermined time intervals as the subject's body width. Taking the skeletal information shown in Figure 3 as an example, the identification unit 223 calculates the distance to the extracted left shoulder LS and right shoulder RS.

[0048] The specific unit 223 then correlates the calculated body width of the subject with the elapsed time. Figure 5 shows a graph of the relationship between the subject's body width and the elapsed time. Figure 5 is an example of a graph showing the relationship between the subject's body width and the elapsed time in this embodiment.

[0049] As shown in Figure 5, when a subject is photographed from the front, the subject initially approaches the camera, so the subject's body width gradually widens over time. Then, when the subject turns around, the subject's body width temporarily narrows, and then widens again. Subsequently, the subject moves away from the camera, so the subject's body width gradually narrows over time. Then, when the subject turns around again, the subject's body width narrows again, and then widens again.

[0050] Here, the absolute value of the percentage change in the subject's body width when the subject is turning (the amount of change in body width over a predetermined time divided by that predetermined time) is greater than the absolute value of the percentage change in the subject's body width when the subject is not turning. Therefore, the identification unit 223 can identify the subject's turning section by calculating the percentage change in the subject's body width.

[0051] Taking the graph shown in Figure 5 as an example, the identification unit 223 identifies two intervals in which the rate of change in the subject's body width is relatively large (the body width narrows) as turning intervals. For example, if the rate of change in body width during a predetermined period (e.g., 5 seconds) is greater than or equal to a predetermined threshold rate (e.g., 20%), the identification unit 223 identifies that predetermined period as a turning interval.

[0052] Furthermore, the identification unit 223 identifies the earlier of the two turning sections as Turn 1, which is Turn 1, and the later of the two turning sections as Turn 2, which is Turn 2.

[0053] Here, the identification unit 223 may calculate the waist width as the subject's body width, either in place of or in addition to the shoulder width. Specifically, the identification unit 223 may identify the positions of the subject's left and right waists by referring to skeletal information and calculate the distance between the left and right waists as the subject's body width. Taking the skeletal information shown in Figure 3 as an example, the identification unit 223 calculates the distance between the extracted left waist LW and right waist RW as the subject's body width.

[0054] Furthermore, in order for the specific unit 223 to easily detect changes in body width, the video is preferably a video that includes the subject captured from the front in its field of view. On the other hand, the video may also be a video that includes the subject captured from the side or rear in its field of view.

[0055] Furthermore, the identification unit 223 may identify the rotation interval based on a value obtained by normalizing the shoulder width using a length that does not change even when the subject rotates (for example, the distance from the shoulder to the waist), rather than using the shoulder width directly. Taking the skeletal information shown in Figure 3 as an example, the identification unit 223 may identify the rotation interval based on a value obtained by normalizing the distance from the left shoulder LS to the right shoulder RS, using at least one of the extracted distance from the left shoulder LS to the left waist LW and the distance from the right shoulder RS ​​to the right waist RW.

[0056] (An example of the process by which the specific unit 223 identifies the walking section) An example of the process by which the identification unit 223 identifies the walking section will be described below.

[0057] The identification unit 223 identifies the walking section in which the subject is seen walking by calculating the distance the subject's feet have moved in the video based on the skeletal information extracted by the extraction unit 222. As an example of a configuration for calculating the distance moved, the identification unit 223 identifies the position of the toes of the subject's left and right feet by referring to the skeletal information, and calculates the distance the toes of the left and right feet have moved in the video at predetermined intervals as the distance the feet have moved.

[0058] An example of the process by which the identification unit 223 calculates the distance traveled will be explained using Figure 6. Figure 6 is a diagram showing an example of the distance traveled calculated by the identification unit 223 in this embodiment. First, the identification unit 223 identifies the right toe tip RT1 and the left toe tip LT1 extracted by the extraction unit 222 in the image PIC1 which includes the subject TP before movement.

[0059] Next, the identification unit 223 identifies the right toe tip RT2 and the left toe tip LT2 extracted by the extraction unit 222 in image PIC2 (in other words, image PIC2 including the subject TP after movement) which is obtained after a predetermined time has elapsed since image PIC1.

[0060] The identification unit 223 then calculates the distance d1 between the right toe RT1 before movement and the right toe RT2 after movement, and the distance d2 between the left toe LT1 before movement and the left toe LT2 after movement. Finally, the identification unit 223 calculates the distance traveled by the left and right feet by dividing the calculated distances d1 and d2 by the predetermined time.

[0061] As an example of how to calculate the distance d1, if the position of the right toe tip RT1 is given by coordinates (x1, y1) and the position of the right toe tip RT2 is given by coordinates (x2, y2), the identification unit 223 calculates the distance d1 using the following equation (1). The identification unit 223 calculates the distance d2 in the same manner.

[0062]

number

[0063] Next, the specific unit 223 correlates the calculated foot movement distance with the elapsed time. Figure 7 shows a graph of the relationship between foot movement distance and elapsed time. Figure 7 is an example of a graph showing the relationship between the subject's foot movement distance and elapsed time in this embodiment.

[0064] As shown in Figure 7, initially, the subject is seated, so the distance the feet move is short. Next, when the subject starts walking, the distance the feet move becomes longer compared to when the subject is seated. Finally, when the subject finishes walking, the distance the feet move becomes shorter again compared to when walking. Therefore, the identification unit 223 identifies the section in which the distance the feet move is relatively longer as the walking section.

[0065] Taking the graph shown in Figure 7 as an example, the identification unit 223 identifies the sections in which the distance traveled by each foot has increased as the right foot walking section and the left foot walking section, respectively. For example, if the distance traveled by each foot is greater than or equal to a predetermined percentage (e.g., 200%) compared to the initial distance traveled in the video (i.e., the distance traveled by the feet while seated) for a predetermined period (e.g., 5 seconds), the identification unit 223 identifies that predetermined period as a walking section.

[0066] The identification unit 223 then identifies the sum of the right-foot walking section and the left-foot walking section (the section from the starting point of the section that starts earlier to the ending point of the section that ends later) as the walking section.

[0067] Here, the identification unit 223 may, in addition to the toes, identify the position of the heel or ankle by referring to skeletal information, and calculate the distance traveled by the heel or ankle as the distance traveled by the foot. For example, using the skeletal information shown in Figure 3, the identification unit 223 may calculate the distance traveled by the foot as the extracted distance traveled by the left heel LH ​​and the right heel RH.

[0068] (An example of the process by which the specific unit 223 identifies the standing-up motion section and the sitting-down motion section.) An example of the process by which the identification unit 223 identifies the standing-up motion section and the sitting-down motion section (hereinafter, the standing-up motion section and the sitting-down motion section are collectively referred to as the "standing-up-and-sitting-down section") will be described.

[0069] The identification unit 223 identifies the standing / sitting section in which the subject is captured standing or sitting by calculating the length of the subject's thigh in the video based on the skeletal information extracted by the extraction unit 222. As an example of a configuration for calculating the length of the thigh, the identification unit 223 identifies the positions of the subject's left and right knees and hips by referring to the skeletal information, and calculates the length from the right knee to the right hip and the length from the left knee to the left hip in the video at predetermined time intervals as the length of the right thigh and the length of the left thigh, respectively. Taking the skeletal information shown in Figure 3 as an example, the identification unit 223 calculates the distance from the extracted right knee RK to the right hip RW and the distance from the extracted left knee LK to the left hip LW as the length of the right thigh and the length of the left thigh, respectively.

[0070] The specific unit 223 then correlates the calculated thigh length with the elapsed time. Figure 8 shows a graph of the relationship between thigh length and elapsed time. Figure 8 is an example of a graph showing the relationship between the thigh length and elapsed time of the subject in this embodiment.

[0071] As shown in Figure 8, initially, the subject is seated, so the length of their thighs is short. Next, when the subject stands up, the length of their thighs changes significantly compared to when they are seated, becoming longer. Subsequently, as the subject moves closer to the camera, the length of their thighs gradually increases over time. Then, after the subject turns around, they move away from the camera, so the length of their thighs gradually decreases over time. Finally, when the subject sits down, the length of their thighs changes significantly compared to when they were walking, becoming shorter. Therefore, the identification unit 223 identifies the section in which the length of the subject's thighs changed relatively significantly as the stand-up-sit-down section.

[0072] Taking the graph shown in Figure 8 as an example, the identification unit 223 identifies two intervals in which the length of the right thigh changes significantly from the seated or walking state as the right leg standing-sitting interval. For example, if the length of the right thigh changes by a predetermined percentage (for example, 20%) or more during a predetermined period (for example, 3 seconds), the identification unit 223 identifies that predetermined period as the right leg standing-sitting interval.

[0073] Furthermore, the identification unit 223 identifies the right leg standing-up-and-sitting section that is earlier in time as the right leg standing-up-and-sitting section, and the right leg standing-up-and-sitting section that is later in time as the right leg sitting-down section. Alternatively, the identification unit 223 identifies the section in which the rate of change in thigh length (the amount of change in thigh length in a predetermined time divided by the predetermined time) is greater than the rate of change in thigh length in a seated state as the right leg standing-up section, and the section in which the rate of change in thigh length is smaller than the rate of change in thigh length during walking as the right leg sitting-down section.

[0074] The identification unit 223 similarly identifies the left leg's standing-to-sitting interval, the left leg's standing-to-sitting movement interval, and the left leg's sitting-to-sitting movement interval.

[0075] The identification unit 223 then identifies the sum of the right leg standing motion period and the left leg standing motion period (the period from the start point of the earlier-starting period of the right leg standing motion period and the end point of the later-ending period) as the standing motion period. The identification unit 223 also identifies the sum of the right leg sitting motion period and the left leg sitting motion period (the period from the start point of the earlier-starting period of the right leg sitting motion period and the end point of the later-ending period) as the sitting motion period.

[0076] Here, the identification unit 223 may identify the sitting motion interval based on a value obtained by normalizing the length of the thigh, using a length that does not change even when the subject performs a standing or sitting motion (for example, the length from the shoulder to the waist), in addition to the length of the thigh, similar to the process of identifying the turning motion interval.

[0077] (An example of the process by which the specific unit 223 generates interval information) An example of the process by which the specific unit 223 generates interval information will be explained using Figure 9. Figure 9 is a diagram showing an example of the process by which the specific unit 223 generates interval information in this embodiment.

[0078] As shown in the top row of Figure 9, the identification unit 223 first determines the length of the video (the "TUG test video" in Figure 9) stored in the memory unit 21.

[0079] Next, the identification unit 223 identifies the turning section 1, turning section 2, and walking section using the method described above (second and third rows from the top in Figure 9). The identification unit 223 then identifies the section where the walking section and turning section 1 overlap as turning section 1, and the section where the walking section and turning section 2 overlap as turning section 2. Furthermore, the identification unit 223 identifies the section of the walking section that is earlier in time than turning section 1 as walking section 1 (fourth row from the top in Figure 9). In addition, the identification unit 223 identifies the section of the walking section between turning section 1 and turning section 2 as walking section 2 (fourth row from the top in Figure 9).

[0080] Next, the identification unit 223 identifies the standing-up motion section and the sitting-down motion section (second row from the bottom in Figure 9). The identification unit 223 then identifies the section that is chronologically before the standing-up motion section as the sitting-down 1 section, which is the sitting-down state at the start of the TUG test. The identification unit 223 also identifies the section that is chronologically after the sitting-down motion section as the sitting-down 2 section, which is the sitting-down state at the end of the TUG test (bottom row in Figure 9).

[0081] As shown in the bottom row of Figure 9, the identification unit 223 generates segment information that identifies each segment within the video in which the subject's actions are captured.

[0082] (An example of processing performed by the evaluation unit 224) Next, we will describe an example of the processing performed by the evaluation unit 224.

[0083] For example, when the evaluation unit 224 evaluates walking speed, since both walk 1 and walk 2 in the TUG test are walks of 3m, the evaluation unit 224 refers to the section information and calculates the walking speed using the time of walk 1 section, the time of walk 2 section, and the distance of 3m.

[0084] As another example, when the evaluation unit 224 evaluates swaying, it calculates the range of motion of the subject's neck and waist by referring to the skeletal information. The evaluation unit 224 then generates an evaluation result indicating that the narrower the range of motion of the subject's neck and waist, the less swaying there is. For example, the evaluation unit 224 generates an evaluation result indicating that the swaying is small when the range of motion of the subject's neck and waist is smaller than a predetermined threshold.

[0085] On the other hand, the evaluation unit 224 generates evaluation results indicating that the greater the range of motion of the subject's neck and waist, the greater the sway. For example, the evaluation unit 224 generates evaluation results indicating that the sway is greater when the range of motion of the subject's neck and waist exceeds a predetermined threshold. Furthermore, the evaluation unit 224 may evaluate the sway by referring to interval information and evaluating the sway for each interval of movement.

[0086] As another example, when the evaluation unit 224 evaluates left-right differences, it calculates the ground contact time of the subject's right and left feet by referring to the skeletal information. The evaluation unit 224 then generates an evaluation result indicating a smaller left-right difference the smaller the difference in ground contact time between the subject's right and left feet. On the other hand, the evaluation unit 224 generates an evaluation result indicating a larger left-right difference the larger the difference in ground contact time between the subject's right and left feet. Similar to the evaluation of unsteadiness, the evaluation unit 224 may also refer to the interval information and evaluate the left-right difference for each interval of movement when evaluating left-right differences.

[0087] (Processing flow in information processing system 100) The processing flow in the information processing system 100 will be explained using Figure 10. Figure 10 is a sequence diagram showing the processing flow in the information processing system 100 according to this embodiment.

[0088] (Step S11) In step S11, the acquisition unit 121 of the information processing device 1 acquires a video image that includes the subject in its field of view via the communication unit 10 or the imaging unit 13. The acquisition unit 121 stores the acquired video image in the storage unit 11.

[0089] (Step S12) In step S12, the output unit 122 outputs the video stored in the storage unit 11 to the information processing device 2 via the communication unit 10.

[0090] (Step S13) In step S13, the acquisition unit 221 of the information processing device 2 acquires the video output from the information processing device 1. The acquisition unit 221 stores the acquired video in the storage unit 21.

[0091] (Step S14) In step S14, the extraction unit 222 extracts the subject's skeletal information from the video stored in the storage unit 21. The extraction unit 222 then stores the extracted skeletal information in the storage unit 21.

[0092] (Step S15) In step S15, the identification unit 223 calculates the subject's body width based on the skeletal information stored in the memory unit 21. An example of how the identification unit 223 calculates the subject's body width is as described above.

[0093] (Step S16) In step S16, the identification unit 223 identifies a turning section within the video footage in which the subject is shown turning, based on the subject's body width calculated in step S15. One example of how the identification unit 223 identifies the turning section is as described above.

[0094] (Step S17) In step S17, the identification unit 223 calculates the distance the subject's feet have moved based on the skeletal information stored in the memory unit 21. An example of how the identification unit 223 calculates the distance moved is as described above.

[0095] (Step S18) In step S18, the identification unit 223 identifies a walking section within the video footage in which the subject is seen walking, based on the distance traveled calculated in step S17. An example of how the identification unit 223 identifies the walking section is as described above.

[0096] (Step S19) In step S19, the identification unit 223 calculates the length of the subject's thigh based on the skeletal information stored in the memory unit 21. An example of how the identification unit 223 calculates the length of the thigh is as described above.

[0097] (Step S20) In step S20, the identification unit 223 identifies a standing-sitting section within the video footage in which the subject is captured standing or sitting, based on the thigh length calculated in step S19. An example of how the identification unit 223 identifies the standing-sitting section is as described above.

[0098] (Step S21) In step S21, the identification unit 223 refers to the turning section, walking section, and standing-sitting section identified in steps S16, S18, and S20, respectively, and generates section information indicating the identified sections. The identification unit 223 stores the generated section information in the storage unit 21. An example of how the identification unit 223 generates section information is as described above.

[0099] (Step S22) In step S22, the evaluation unit 224 refers to the skeletal information and interval information stored in the memory unit 21 and performs an evaluation of the subject's movements. The evaluation unit 224 stores the evaluation results in the memory unit 21. An example of how the evaluation unit 224 performs the evaluation is as described above.

[0100] (Step S23) In step S23, the output unit 225 outputs the evaluation results stored in the storage unit 21 to the information processing device 1 via the communication unit 20.

[0101] (Step S24) In step S24, the acquisition unit 121 of the information processing device 1 acquires the evaluation results output from the information processing device 2 via the communication unit 10. The acquisition unit 121 stores the acquired evaluation results in the storage unit 11.

[0102] (Step S25) In step S25, the display unit 123 displays the evaluation results stored in the memory unit 11 on the display 14.

[0103] (Example of a display screen) An example of a display screen to be shown on the display 14 in step S25 described above will be explained with reference to Figure 11. Figure 11 is a diagram showing an example of a display screen to be shown on the display 14 in this embodiment.

[0104] As shown in Figure 11, the display screen that the display unit 123 displays on the display 14 in step S25 includes the evaluation results ER1. The evaluation results ER1 includes each evaluation item and the evaluation result for each evaluation item. Therefore, the display unit 123 can present the evaluation results to the user.

[0105] (An example of the process for playing the video MV1) Furthermore, as shown in Figure 11, the display screen that the display unit 123 displays on the display 14 in step S25 may include a display area DR1 that displays a video MV1 in which the subject is included in the field of view. For example, the display unit 123 may display the video MV1 stored in the storage unit 11 in the display area DR1. Also, as shown in Figure 11, the display unit 123 may display a button P_BT1, which is a playback start operator for receiving operation input from the user to start playback of the video MV1. In this case, the display unit 123 first displays an arbitrary still image included in the video MV1 (for example, a still image at the start of the video MV1) and the button P_BT1.

[0106] Next, when the reception unit 124 obtains position information indicating the location where contact with the display 14 was detected, the reception unit 124 determines whether the position indicated by the position information is included in the display range of button P_BT1 (in other words, whether or not button P_BT1 was pressed by the user).

[0107] If the location information is determined to be included in the position where button P_BT1 is displayed (in other words, if it is determined that the user has pressed button P_BT1), the reception unit 124 stores operation input information indicating that button P_BT1 has been pressed in the storage unit 11.

[0108] Next, the processing execution unit 125 refers to the operation input information stored in the storage unit 11, and if the operation input information indicates that button P_BT1 has been pressed, it starts playing the video MV1.

[0109] The process in which the reception unit 124 acquires location information, generates operation input information indicating operation input from the user, and the processing execution unit 125 refers to this operation input information to determine whether or not it has received any operation input from the user is also described as "the processing execution unit 125 determines whether or not it has received any operation input from the user."

[0110] Furthermore, as shown in Figure 11, the display unit 123 may also include a playback bar P_BA1 on the display screen of the display 14, which indicates which point in the moving image MV1 is being played.

[0111] (An example of a process that plays video MV1 from the position where the subject starts a predetermined action) The processing execution unit 125 may be configured to play the video MV1 from the position where the subject starts a predetermined action. In this case, in step S23 described above, the output unit 225 of the information processing device 2 outputs the section information stored in the storage unit 21 to the information processing device 1 in addition to the evaluation result. When the acquisition unit 121 of the information processing device 1 acquires the section information in addition to the evaluation result, it stores the section information in the storage unit 21.

[0112] Next, the display unit 123 refers to the section information stored in the memory unit 21 and further includes selection controls on the display screen of the display 14 for playing the video MV1 from the position where the subject starts a predetermined action. In the diagram shown in Figure 11, the display unit 123 further includes on the display screen of the display 14 buttons such as button M_BT1, which is a selection control for playing the video MV1 from the position where the subject starts standing up; button M_BT2, which is a selection control for playing the video MV1 from the position where the subject starts walking 1; and button M_BT3, which is a selection control for playing the video MV1 from the position where the subject starts turning 1.

[0113] In this case, the processing execution unit 125 determines whether it has received an operation input from the user to play the video MV1 from the position where the subject starts a predetermined action. For example, the processing execution unit 125 determines whether it has received an operation input from the user to the button M_BT3 to play the video MV1 from the position where the subject starts turning 1. If it determines that it has received an operation input to the button M_BT3 from the user, the processing execution unit 125 further determines whether it has received an operation input from the user to play the video MV1 (in other words, whether it has received an operation input from the user to the button P_BT1 which is the operation for starting playback). If the processing execution unit 125 determines that it has received an operation input from the user to play the video MV1, it plays the video MV1 from the position where the subject starts turning 1.

[0114] Here, if the processing execution unit 125 determines that it has received an input from the user to the button M_BT3 for playing the video MV1 from the position where the subject starts turning 1, the processing execution unit 125 may change the display mode of the button M_BT3, as shown in Figure 11.

[0115] (Another example of a process that plays video MV1 from the position where the subject starts a predetermined action) Furthermore, when the acquisition unit 121 acquires section information, the display unit 123 may indicate the point in time when the subject starts a predetermined action on the playback bar P_BA1, and may also include selection controls on the display screen of the display 14 for selecting the playback start position. In the diagram shown in Figure 11, the display unit 123 further includes on the display screen of the display 14 a flag FL_BT1 as a selection control for playing the video MV1 from the position where the subject starts standing, and a flag FL_BT2 as a selection control for playing the video MV1 from the position where the subject starts walking 1, above the playback bar P_BA1.

[0116] In this case, the processing execution unit 125 determines whether or not it has received an operation input from the user for a flag. For example, if it determines that it has received an operation input from the user for the flag FL_BT2, which is used to play the video MV1 from the position where the subject starts walking 1, the processing execution unit 125 further determines whether or not it has received an operation input from the user for the button P_BT1, which is the operation for starting playback. If the processing execution unit 125 determines that it has received an operation input from the user for the button P_BT1, it plays the video MV1 from the position where the subject starts walking 1.

[0117] Thus, in the information processing system 100 of this embodiment, the information processing device 2 includes an acquisition unit 221 that acquires a moving image MV1 that includes the subject in its field of view, an extraction unit 222 that extracts the subject's skeletal information from the moving image MV1, and an identification unit 223 that identifies a turning section within the section included in the moving image MV1 in which the subject is shown turning, based on the skeletal information. Therefore, the information processing device 2 can evaluate the subject's movements in the turning section, and thus can evaluate the subject's ability to move their own body in detail.

[0118] [Embodiment 2] Other embodiments of the present invention will be described below. For the sake of clarity, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0119] (Overview of Information Processing System 200) In the information processing system 200 according to this embodiment, the information processing device 1 is configured to simultaneously start playback of multiple video footage from one of a plurality of pre-set playback start positions. An example of a plurality of pre-set playback start positions is the position in which a subject included as a subject in each of the plurality of video footage begins a predetermined action in each of the plurality of video footage. As an example, the information processing device 1 simultaneously starts playback of each of the plurality of video footage from the position in which the subject begins to turn. As another example, the information processing device 1 simultaneously starts playback of one video from the position in which the subject begins to turn, and another video from the position in which the subject begins to walk.

[0120] In addition to the multiple videos containing the subject of the TUG test described above, the information processing system 200 can also use the following as multiple videos to be played back simultaneously: multiple videos for analyzing the subject's gait, multiple videos capturing the subject playing sports (e.g., swimming), multiple videos capturing animals (e.g., mice in animal experiments), and multiple videos used for presentations (e.g., icebreakers).

[0121] Here, the configuration and functions of the information processing device 2 are as described in the embodiments described above, so we will omit further explanation.

[0122] (Configuration of Information Processing Device 1) Figure 12 shows the configuration of the information processing system 200 according to this embodiment.

[0123] As shown in Figure 12, the information processing device 1 comprises a communication unit 10, a storage unit 11, a control unit 12, an imaging unit 13, and a display 14. The communication unit 10, imaging unit 13, and display 14 are the same as those in the embodiments described above, so their description is omitted.

[0124] The memory unit 11 stores data that the control unit 12 references. For example, the memory unit 11 stores multiple video images, along with evaluation results and interval information associated with those video images. The evaluation results and interval information are as described above.

[0125] (Functions of the control unit 12) The control unit 12 controls each component of the information processing device 1. As shown in Figure 12, the control unit 12 also functions as an acquisition unit 121, an output unit 122, a display unit 123, a reception unit 124, and a processing execution unit 125.

[0126] The acquisition unit 121 acquires data supplied from the communication unit 10 and the imaging unit 13. As an example, in addition to the video and evaluation results described in the above embodiment, the acquisition unit 121 acquires section information. The acquisition unit 121 associates the acquired data with each other and stores it in the storage unit 11. An example of the processing performed by the acquisition unit 121 will be described later.

[0127] The output unit 122 outputs data to the information processing device 2 via the communication unit 10. For example, the output unit 122 outputs a video image stored in the storage unit 11 to the information processing device 2 via the communication unit 10.

[0128] The display unit 123 supplies the image to be displayed to the display 14 and displays the image on the display 14. As an example, the display unit 123 displays a display screen on the display 14 that includes a display area for displaying each of the multiple video images stored in the storage unit 11, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the multiple video images, and a playback start operator for starting playback of the multiple video images simultaneously. Examples of the display screen displayed by the display unit 123 and examples of the processes it executes will be described later.

[0129] The reception unit 124 receives operation input from the user. For example, the reception unit 124 receives operation input via the selection operator and the playback start operator. The reception unit 124 stores operation input information indicating the received operation input in the storage unit 11.

[0130] The processing execution unit 125 executes processing in response to user operations. For example, the processing execution unit 125 refers to the operation input information stored in the storage unit 11, and when the playback start operator is operated, it simultaneously starts playback of multiple video images from the playback start position selected by the selection operator. Examples of processing performed by the processing execution unit 125 will be described later.

[0131] (An example of the process by which the acquisition unit 121 stores data in the storage unit 11) An example of the process by which the acquisition unit 121 stores data in the storage unit 11 will be described below.

[0132] The acquisition unit 121 associates the video image with the evaluation results and section information and stores them in the storage unit 11.

[0133] One example is a configuration in which the acquisition unit 121 attaches identification information as tag information to the video image acquired from the imaging unit 13 to distinguish the video image from other video images.

[0134] In this case, when the acquisition unit 121 acquires a video image supplied from the imaging unit 13, it attaches identification information as tag information to the video image and stores it in the storage unit 11. The video image with tag information is output to the information processing device 2 by the output unit 122. When the acquisition unit 221 of the information processing device 2 acquires a video image with tag information from the information processing device 1, the identification unit 223 and the evaluation unit 224 generate section information and evaluation results, respectively, from the video image. The process by which the identification unit 223 and the evaluation unit 224 generate section information and evaluation results, respectively, is as described above.

[0135] Next, the identification unit 223 and the evaluation unit 224 each attach the same identification information as the tag information attached to the video footage referenced to generate the generated section information and evaluation results as tag information. Then, the output unit 225 outputs the section information and evaluation results with the attached tag information to the information processing device 1.

[0136] Next, the acquisition unit 121 of the information processing device 1 acquires the section information and evaluation results, which are tagged with tag information, output from the information processing device 1. Then, the acquisition unit 121 associates the acquired section information and evaluation results with a video image stored in the storage unit 11 that has the same tag information as the section information and evaluation results, and stores it in the storage unit 11.

[0137] Furthermore, the acquisition unit 121 may, in addition to the identification information, further attach subject identification information to the video as tag information to distinguish the subjects included as subjects from other subjects. In this case as well, the identification unit 223 and evaluation unit 224 of the information processing device 2 attach the same subject identification information as the tag information attached to the video referenced to generate the section information and evaluation results, respectively, to the generated section information and evaluation results. The output unit 225 then outputs the section information and evaluation results with the attached tag information to the information processing device 1. The acquisition unit 121 of the information processing device 1 stores the acquired section information and evaluation results in the storage unit 11, associating them with the video images stored in the storage unit 11 that have the same tag information as the tag information attached to the acquired section information and evaluation results.

[0138] (Another example of the process by which the acquisition unit 121 stores data in the storage unit 11) Another example is a configuration in which the acquisition unit 121 acquires video footage, section information, and evaluation results from the information processing device 2 via the communication unit 10.

[0139] In this case, the output unit 225 of the information processing device 2 outputs the interval information and evaluation results generated by the identification unit 223 and the evaluation unit 224, respectively, and the video footage referenced to generate the interval information and evaluation results, to the information processing device 1.

[0140] Next, the acquisition unit 121 of the information processing device 1 acquires the video, evaluation results, and section information output from the information processing device 2, associates the acquired video, evaluation results, and section information, and stores them in the storage unit 11.

[0141] Even if the acquisition unit 121 acquires the video, section information, and evaluation results from the information processing device 2 via the communication unit 10, the video, section information, and evaluation results may be configured to have the aforementioned tag information attached.

[0142] (An example of the process by which the display unit 123 displays the screen.) An example of the process by which the display unit 123 displays the display screen will be described below.

[0143] The display unit 123 first selects a number of videos to display from among the videos stored in the storage unit 11. The number of videos that the display unit 123 displays is not particularly limited, but examples include a number of videos each instructed by the user, the video with the most recent date and time stored in the storage unit 11, and a video with tag information indicating the same subject as the subject indicated by the tag information attached to that video (in other words, the latest video and another video that includes the same subject as the subject included as the subject in the latest video).

[0144] Next, the display unit 123 refers to the section information associated with each of the selected video images and sets multiple playback start positions. For example, if the subject performs the TUG test as described in the embodiment described above, the display unit 123 refers to the section information and sets the start times of the video images for the multiple predetermined actions performed by the subject, namely "standing up," "walking 1," "turning 1," "walking 2," "turning 2," and "sitting down," as multiple playback start positions.

[0145] Finally, the display unit 123 displays the selected multiple videos in the display area included in the display screen. Furthermore, the display unit 123 displays a display screen that includes a selection operator for selecting one of the set multiple playback start positions, and a playback start operator for simultaneously starting playback of the multiple videos from the playback start position selected by the selection operator.

[0146] Here, the multiple predetermined actions performed by the subject may include not only walking, turning, standing up, and sitting down, but also tripping and unsteadiness. In other words, the actions identified by the identification unit 223 of the information processing device 2 may be considered as the predetermined actions performed by the subject. By referring to the section information indicating the section in which the action identified by the information processing device 2 was performed, the display unit 123 can identify which predetermined action is included in the moving image.

[0147] (Example of a display screen) An example of a display screen shown by the display unit 123 will be explained using Figure 13. Figure 13 is a diagram showing an example of a display screen shown by the display unit 123 in this embodiment.

[0148] As shown in Figure 13, the display screen displayed by the display unit 123 includes display areas DR2 and DR3, which display each of the multiple video images (video image MV2 and video image MV3) selected by the display unit 123. The display screen displayed by the display unit 123 also includes selection controls such as buttons M_BT4 to M_BT8 for selecting one of several pre-set playback start positions (the start time of each of the multiple actions performed by the subject), and a playback start control such as button P_BT2 for simultaneously starting playback of multiple video images.

[0149] Furthermore, the display unit 123 may also be configured to include a playback bar P_BA2 indicating which point in the video MV2 is being played, and a playback bar P_BA3 indicating which point in the video MV3 is being played, as shown in Figure 13.

[0150] Furthermore, as shown in Figure 13, the display unit 123 may also be configured to include, as selection controls, flags FL_BT3 to FL_BT6 on the display screen, which indicate the point in time when the subject starts a predetermined action in playback bar P_BA2 and playback bar P_BA3, respectively, by referring to the section information associated with the selected video MV2 and video MV3. For example, flag FL_BT3 indicates the start time of the subject's "turn 2" included in video MV2, and flag FL_BT5 indicates the start time of the subject's "standing up" included in video MV3.

[0151] Furthermore, as shown in Figure 13, the display unit 123 may be configured to include evaluation results ER2 and ER3 associated with the selected video MV2 and video MV3, respectively, on the display screen. In this case, when the evaluation unit 224 of the information processing device 2 evaluates the operation of each section, the display unit 123 may be configured to display the evaluation result corresponding to the section of the video being played back. For example, the display unit 123 may be configured to display the evaluation result of one rotation section during the period in which one rotation section of the video is being played back.

[0152] (An example of processing performed by the processing execution unit 125) An example of the processing performed by the processing execution unit 125 will be described below.

[0153] For example, in the display screen shown in Figure 13, when the reception unit 124 receives an operation input indicating that buttons M_BT5, M_BT8, and P_BT2 have been pressed, the reception unit 124 stores the operation input information indicating the operation input in the storage unit 11. The processing execution unit 125 refers to the operation input information stored in the storage unit 11 and simultaneously starts playback of video MV2 and video MV3 from the playback start position of "Walking 1" in video MV2, which is the playback start position corresponding to button M_BT5, and the playback start position of "Turning 1" in video MV3, which is the playback start position corresponding to button M_BT8.

[0154] As another example, if the operation input information stored in the memory unit 11 indicates that flags FL_BT3, FL_BT5, and button P_BT2 have been pressed, the processing execution unit 125 simultaneously starts playback of video MV2 and video MV3 from the playback start position of "Swivel 2" in video MV2, which corresponds to the playback start position of flag FL_BT3, and the playback start position of "Standing Up" in video MV3, which corresponds to the playback start position of flag FL_BT5.

[0155] (Processing flow in Information Processing System 200) The processing flow in the information processing system 200 will be explained using Figure 14. Figure 14 is a sequence diagram showing the processing flow in the information processing system 200 according to this embodiment.

[0156] (Steps S11-S12) Steps S11 to S12, in which the acquisition unit 121 of the information processing device 1 acquires a moving image and the output unit 122 outputs the moving image to the information processing device 2, are as described above and therefore will not be explained further.

[0157] (Steps S13 to S22) Steps S13 to S20, in which the acquisition unit 221 of the information processing device 2 acquires a moving image, the extraction unit 222 extracts the subject's skeletal information from the moving image, and the identification unit 223 identifies the turning section, walking section, and standing-sitting section based on the skeletal information, are as described above and therefore will not be explained.

[0158] Furthermore, steps S21 to S22, in which the identification unit 223 generates section information indicating the identified section and the evaluation unit 224 performs an evaluation of the subject's movements by referring to the skeletal information and section information, are as described above and will therefore not be explained.

[0159] (Step S31) In step S31, the output unit 225 outputs the evaluation results and section information stored in the storage unit 21 to the information processing device 1 via the communication unit 20. Here, as described above, the output unit 225 may also be configured to output the video footage referenced to generate the evaluation results and section information, in addition to the evaluation results and section information.

[0160] (Step S32) In step S32, the acquisition unit 121 of the information processing device 1 acquires the evaluation results and section information output from the information processing device 2 via the communication unit 10. The acquisition unit 121 stores the acquired evaluation results and section information in the storage unit 11, associating them with the video footage referenced to generate the evaluation results and section information. The process by which the acquisition unit 121 stores the acquired data in the storage unit 11 is as described above.

[0161] (Step S33) In step S33, the display unit 123 displays a display screen that includes a display area for displaying each of the multiple video images, a selection operator for selecting one of a set of preset playback start positions for each of the multiple video images, and a playback start operator for starting playback of the multiple video images simultaneously. The process by which the display unit 123 displays the display screen and an example of the display screen to be displayed are as described above.

[0162] (Step S34) In step S34, the reception unit 124 determines whether or not it has received operation input via the selection operator and the playback start operator.

[0163] If, in step S34, the reception unit 124 determines that it has not received any operation input via the selection operator and the playback start operator (step S34: NO), the reception unit 124 executes the process of step S34 again.

[0164] (Step S35) In step S34, the reception unit 124 determines that it has received operation input via the selection operator and the playback start operator (step S34: YES), and if the playback start operator is operated, the processing execution unit 125 simultaneously starts playback of multiple video images from the playback start position selected by the selection operator. The processing executed by the processing execution unit 125 is as described above.

[0165] As described above, in the information processing system 200 of this embodiment, the information processing device 1 includes a display unit 123 that displays a display screen including display areas DR2 and DR3 for displaying each of a plurality of moving images MV2 and moving images MV3, buttons M_BT4 to M_BT8 and flags FL_BT3 to FL_BT6 for selecting one of a plurality of preset playback start positions for each of the moving images MV2 and moving images MV3, and a button P_BT2 for simultaneously starting playback of the moving images MV2 and moving images MV3, a reception unit 124 that receives operation input via buttons M_BT4 to M_BT8 and flags FL_BT3 to FL_BT6 and button P_BT2, and a processing execution unit 125 that, when button P_BT2 is operated, simultaneously starts playback of the moving images MV2 and moving images MV3 from the playback start position selected by buttons M_BT4 to M_BT8 and flags FL_BT3 to FL_BT6.

[0166] Therefore, the information processing device 1 can simultaneously play back both the video MV2 and the video MV3 from any of a set of preset playback start positions, allowing for a more favorable comparison of multiple videos.

[0167] [Embodiment 3] Other embodiments of the present invention will be described below. For the sake of clarity, components having the same function as those described in the above embodiments will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0168] (Configuration of Information Processing Device 3) The information processing device 3 according to this embodiment has the configuration of the information processing device 1 and information processing device 2 in the information processing system 100 and information processing system 200 described above. Figure 15 is a block diagram showing the configuration of the information processing device 3 according to this embodiment.

[0169] As shown in Figure 15, the information processing device 3 comprises an imaging unit 13, a display 14, a storage unit 31, and a control unit 32. The imaging unit 13 and the display 14 are the same as those in the embodiments described above, so their description is omitted.

[0170] Similar to the storage units 11 and 21 described above, the storage unit 31 stores data that the control unit 32 references. Examples of data stored in the storage unit 31 include video footage, skeletal information, interval information, evaluation results, and operation input information. The video footage, skeletal information, interval information, evaluation results, and operation input information are as described above.

[0171] (Functions of the control unit 32) The control unit 32 controls each component of the information processing device 3. As shown in Figure 15, the control unit 32 also functions as an acquisition unit 121, a display unit 123, a reception unit 124, a processing execution unit 125, an extraction unit 222, a identification unit 223, and an evaluation unit 224.

[0172] The acquisition unit 121 acquires data supplied from the imaging unit 13. The acquisition unit 121 also associates the acquired data with each other and stores it in the storage unit 11. As an example, the acquisition unit 121 associates the video, evaluation results, and section information with each other and stores them in the storage unit 11. The process by which the acquisition unit 121 associates the video, evaluation results, and section information with each other and stores them in the storage unit 11 is as described above.

[0173] The display unit 123 supplies the image to be displayed to the display 14 and displays the image on the display 14. As an example, the display unit 123 displays a video stored in the storage unit 11 on the display 14. As another example, the display unit 123 displays a display screen on the display 14 that includes a display area for displaying each of the multiple video stored in the storage unit 11, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the multiple video, and a playback start operator for starting playback of the multiple video simultaneously. The processing performed by the display unit 123 is as described above.

[0174] The reception unit 124 receives operation input from the user. For example, the reception unit 124 receives operation input via the selection operator and the playback start operator. The reception unit 124 stores operation input information indicating the received operation input in the storage unit 31.

[0175] The processing execution unit 125 executes processing in response to user operations. For example, the processing execution unit 125 refers to the operation input information stored in the storage unit 31 and plays back the video when the playback start operator is operated. As another example, the processing execution unit 125 refers to the operation input information stored in the storage unit 31 and, when the playback start operator is operated, simultaneously starts playing back multiple videos from the playback start position selected by the selection operator. The processing executed by the processing execution unit 125 is as described above.

[0176] The extraction unit 222 applies skeletal extraction processing to the video image and extracts skeletal information indicating the skeleton of the subject included in the video image. For example, the extraction unit 222 acquires the video image stored in the storage unit 31 and extracts the subject's skeletal information from the video image. The extraction unit 222 stores the extracted skeletal information in the storage unit 31.

[0177] The identification unit 223 identifies a section within the video where the subject is performing a predetermined action. For example, the identification unit 223 obtains the video and skeletal information extracted by the extraction unit 222 from the storage unit 31, and based on the skeletal information, identifies a turning section within the video in which the subject is shown turning. Alternatively, the identification unit 223 may identify at least one of the following based on the skeletal information: a walking section in which the subject is shown walking, a standing section in which the subject is shown standing up, and a sitting section in which the subject is shown sitting down. The identification unit 223 supplies the section information indicating the identified section to the acquisition unit 121. The processing performed by the identification unit 223 is as described above.

[0178] The evaluation unit 224 evaluates the movements of the subject included as a subject in the video. For example, the evaluation unit 224 obtains the skeletal information extracted by the extraction unit 222 and the section information identified by the identification unit 223 from the storage unit 31, and evaluates the following items regarding the subject's movements by referring to the skeletal information and section information. • Walking speed • Unsteadiness • Left-right difference ·rhythm • Duration ·comprehensive evaluation The evaluation unit 224 supplies the evaluation results to the acquisition unit 121. The processing performed by the evaluation unit 224 is as described above.

[0179] Thus, the information processing device 3 according to this embodiment has the configurations of the information processing device 1 and the information processing device 2 in the information processing system 100 and the information processing system 200 described above. Specifically, the information processing device 3 includes an acquisition unit 121 that acquires a moving image that includes the subject in the field of view, an extraction unit 222 that extracts the subject's skeletal information from the moving image, and an identification unit 223 that identifies a turning section within the section included in the moving image in which the subject is shown turning, based on the skeletal information. Therefore, the information processing device 3 can evaluate the subject's movements in the turning section, and thus can evaluate the subject's ability to move their own body in detail.

[0180] Furthermore, the information processing device 3 includes a display unit 123 that displays a display screen including a display area DR2 and a display area for displaying each of the multiple video images, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the multiple video images, and a playback start operator for simultaneously starting playback of the multiple video images, a reception unit 124 that receives operation input via the selection operator and the playback start operator, and a processing execution unit 125 that, when the playback start operator is operated, simultaneously starts playback of the multiple video images from the playback start position selected by the selection operator.

[0181] Therefore, the information processing device 3 can simultaneously play back each of the multiple video images from any of the pre-set playback start positions, allowing for a more favorable comparison of the multiple video images.

[0182] [Examples of implementation using software] The functions of the information processing devices 1, 2, and 3 (hereinafter referred to as "devices") are programs that cause the devices to function as computers, and these programs can be realized by programs that cause each control block of the devices (particularly the parts included in the control units 12, 22, and 32) to function as computers.

[0183] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., memory) as hardware for executing the program. By executing the program using this control device and storage device, the functions described in each of the embodiments are realized.

[0184] The above program may be recorded on one or more computer-readable recording media, not temporary ones. These recording media may or may not be provided by the above device. In the latter case, the program may be supplied to the above device via any wired or wireless transmission medium.

[0185] Furthermore, some or all of the functions of each of the above control blocks can also be realized by logic circuits. For example, an integrated circuit in which logic circuits functioning as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

[0186] (Physical configuration of information processing device 1 and information processing device 2) Figure 16 is a block diagram illustrating the physical configuration of computers used as information processing device 1 and information processing device 2.

[0187] As shown in Figure 16, the information processing device 1 can be configured as a computer comprising a bus 110, a processor 101, a main memory 102, an auxiliary memory 103, a communication interface 104, a display panel 105, a touchpad 106, and a camera 107. The processor 101, main memory 102, auxiliary memory 103, communication interface 104, display panel 105, touchpad 106, and camera 107 are connected to each other via the bus 110.

[0188] The information processing device 2 can also be configured as a computer, for example, as shown in Figure 16, which includes a bus 210, a processor 201, a main memory 202, an auxiliary memory 203, and a communication interface 204. The processor 201, the main memory 202, the auxiliary memory 203, and the communication interface 204 are connected to each other via the bus 210.

[0189] Processors 101 and 201 can be, for example, a CPU (Central Processing Unit), a microprocessor, a digital signal processor, a microcontroller, or a combination thereof.

[0190] For example, semiconductor RAM (random access memory) can be used as the main memory 102 and the main memory 202.

[0191] Auxiliary memory 103 and auxiliary memory 203 can be, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof. Auxiliary memory 103 and auxiliary memory 203 each store a program that causes the processor 101 to execute the operations of the information processing device 1 and a program that causes the processor 201 to execute the operations of the information processing device 2, respectively. The processor 101 expands the program stored in auxiliary memory 103 onto the main memory 102 and executes each instruction contained in the expanded program. Similarly, the processor 201 expands the program stored in auxiliary memory 203 onto the main memory 202 and executes each instruction contained in the expanded program.

[0192] Communication interface 104 and communication interface 204 are interfaces that connect to network N.

[0193] In this example, the processor 101 and the communication interface 104 are examples of hardware elements that realize the control unit 12 and the communication unit 10. Also, the main memory 102 and the auxiliary memory 103 are examples of hardware elements that realize the storage unit 11.

[0194] Furthermore, the display panel 105 is an example of a hardware element that implements the display panel 141. The touchpad 106 is an example of a hardware element that implements the touchpad 142. The camera 107 is an example of a hardware element that implements the imaging unit 13.

[0195] Furthermore, the processor 201 and the communication interface 204 are examples of hardware elements that realize the control unit 22 and the communication unit 20. Also, the main memory 202 and the auxiliary memory 203 are examples of hardware elements that realize the storage unit 21.

[0196] (Physical configuration of information processing device 3) Figure 17 is a block diagram illustrating the physical configuration of a computer used as an information processing device 3. As shown in Figure 17, the information processing device 3 can be configured by a computer comprising a bus 310, a processor 301, a main memory 302, an auxiliary memory 303, a display panel 304, a touchpad 205, and a camera 306. The processor 301, main memory 302, auxiliary memory 303, display panel 304, touchpad 305, and camera 306 are connected to each other via the bus 310.

[0197] The processor 301 can be, for example, a CPU, a microprocessor, a digital signal processor, a microcontroller, or a combination thereof.

[0198] For the main memory 302, for example, semiconductor RAM can be used.

[0199] For example, the auxiliary memory 303 may be flash memory, HDD, SSD, or a combination thereof. The auxiliary memory 103 stores a program that causes the processor 301 to execute the operations of the information processing device 3 described above. The processor 301 loads the program stored in the auxiliary memory 303 onto the main memory 302 and executes each instruction contained in the loaded program.

[0200] In this example, the processor 301 is an example of a hardware element that implements the control unit 32. The main memory 302 and auxiliary memory 303 are also examples of hardware elements that implement the storage unit 31.

[0201] Furthermore, the display panel 304 is an example of a hardware element that implements the display panel 141. The touchpad 305 is an example of a hardware element that implements the touchpad 142. The camera 307 is an example of a hardware element that implements the imaging unit 13.

[0202] Furthermore, each process described in the above embodiments may be performed by AI (Artificial Intelligence). In this case, the AI ​​may operate on the control device described above, or it may operate on other devices (for example, an edge computer or a cloud server).

[0203] 〔summary〕 An information processing device according to one aspect of each embodiment includes a display unit that displays a display screen including a display area for displaying each of a plurality of moving images, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the plurality of moving images, and a playback start operator for simultaneously starting playback of the plurality of moving images, a reception unit that receives operation input via the selection operator and the playback start operator, and a processing execution unit that, when the playback start operator is operated, simultaneously starts playback of the plurality of moving images from the playback start position selected by the selection operator.

[0204] According to the above configuration, multiple video images can be compared more effectively.

[0205] In an information processing device according to one aspect of each embodiment, the playback start position is preferably the position in which a subject included as a subject in each of the plurality of moving images starts a predetermined action in each of the plurality of moving images.

[0206] With the above configuration, playback can be started simultaneously from the position where a predetermined operation begins in each of the multiple video images.

[0207] In an information processing device according to one aspect of each embodiment, it is preferable that the predetermined operation includes walking, turning, standing up, sitting down, stumbling, and unsteadiness of the subject.

[0208] With the above configuration, playback of each of the multiple video images can be started from the moment the subject performs any of the following actions: walking, turning, standing, sitting, tripping, or unsteadiness.

[0209] In an information processing device according to one aspect of each embodiment, the display unit further displays on the display screen a playback bar indicating which point in the moving image is being played and a flag indicating the point in time when the subject starts the predetermined action on the playback bar, and the selection operator is preferably the flag.

[0210] With the above configuration, by operating the playback bar, playback can be started simultaneously from a predetermined starting position for each of multiple video clips.

[0211] In an information processing device according to one aspect of each embodiment, the selection operator is preferably a button displayed on the display screen.

[0212] With the above configuration, by operating a button, playback can be started simultaneously from a predetermined position in each of multiple video clips.

[0213] An information processing method according to one aspect of each embodiment includes a display step of displaying a display screen that includes a display area for displaying each of a plurality of moving images, a selection operator for selecting one of a plurality of pre-set playback start positions for each of the plurality of moving images, and a playback start operator for simultaneously starting playback of the plurality of moving images; a reception step of receiving operation input via the selection operator and the playback start operator; and a processing execution step of simultaneously starting playback of the plurality of moving images from the playback start position selected by the selection operator when the playback start operator is operated.

[0214] According to the above configuration, multiple video images can be compared more effectively.

[0215] A program according to one aspect of each embodiment causes a computer to function as the display unit, the reception unit, and the processing execution unit.

[0216] According to the above configuration, multiple video images can be compared more effectively.

[0217] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0218] 1, 2, 3 Information Processing Devices 10, 20 Communications Department 11, 21, 31 Storage section 12, 22, 32 Control Unit 13 Imaging Unit 14 displays 100, 200 Information Processing Systems 121, 221 Acquisition Department 122, 225 Output section 123 Display section 124 Reception Department 125 Processing Execution Unit 222 Extraction part 223 Specific part 224 Evaluation Department

Claims

1. A display unit that displays a display screen including a display area for displaying each of a plurality of video images, a selection operator that can be operated independently for each of the plurality of video images for selecting one of a plurality of pre-set playback start positions for each of the plurality of video images, and a playback start operator for starting playback of the plurality of video images simultaneously. A receiving unit that receives operation input via the selection operator and the playback start operator, When the playback start operator is operated, a processing execution unit simultaneously starts playback of the multiple video images from the playback start position selected by the selection operator. Equipped with, The playback start position is the position in which a subject included as a subject in each of the plurality of video images begins a predetermined action in each of the plurality of video images. The display unit further displays on the display screen a playback bar indicating which point in the video is being played, and a flag indicating the point in time when the subject begins the predetermined action on the playback bar. The selection operator is the flag. Information processing device.

2. The aforementioned predetermined actions include walking, turning, standing up, sitting down, stumbling, and unsteadiness by the subject. The information processing apparatus according to claim 1.

3. The aforementioned selection operator is a button displayed on the display screen. The information processing apparatus according to claim 1 or 2.

4. A display step that displays a display screen including a display area for displaying each of a plurality of video images, a selection operator that can be operated independently for each of the plurality of video images for selecting one of a plurality of pre-set playback start positions for each of the plurality of video images, and a playback start operator for starting playback of the plurality of video images simultaneously. A reception step that receives operation input via the selection operator and the playback start operator, When the playback start operator is operated, the process execution step is to simultaneously start playback of the multiple video images from the playback start position selected by the selection operator. Includes, The playback start position is the position in which a subject included as a subject in each of the plurality of video images begins a predetermined action in each of the plurality of video images. In the display step, a playback bar indicating which point in the video is being played, and a flag indicating the point in time when the subject begins the predetermined action are further displayed on the display screen. The selection operator is the flag. Information processing methods.

5. A program that causes the computer to function as the display unit, the reception unit, and the processing execution unit according to claim 1.

Images