Information processing program, information processing method, and information processing device

Abstract

This invention improves recognition accuracy if recognizing a motion to be recognized with respect to a target, in a plurality of frame images included in a video. This information processing program causes a computer to execute processes for: identifying, among a plurality of time series image data pieces, as a posture of the target, first image data which includes a posture to be taken when a target starts a motion to be recognized and second image data which includes a posture to be taken in the midst of the motion to be recognized being executed by the target; and when recognizing the motion to be recognized on the basis of such two types of postures of the target, if the plurality of time series image data pieces transition from the first image data to the second image data and further transition to image data other than the first or second image data, recognizing that the target was executing the motion to be recognized during a period from the first image data to the second image data.

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] A recognition technique is known in which an object is captured in a video and the posture of the object is identified in each frame image from the object's skeletal information. This recognition technique allows the recognition and extraction of a target movement from multiple frame images by predefining, for example, the posture of the object at the start of the target movement, the posture of the object while performing the movement, and the posture of the object at the end of the movement. Note that a movement here refers to a series of movements consisting of a sequence of multiple postures of the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 116495 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when attempting to recognize the motion of a recognition target using the above method, if the target cannot assume one of the predefined postures, the recognition accuracy decreases.

[0005] In one aspect, an object is to improve the recognition accuracy when recognizing the motion of a target object in a plurality of frame images included in a moving image. [Means for solving the problem]

[0006] According to one aspect, the information processing program comprises: From multiple time-series image data, the posture of the object is determined as the posture that the object should take when starting the recognition target movement.First image data including and the posture that the object should take while performing the action to be recognized. but Included Second Image data and Identify the Recognize the movement of the object based on two types of postures of the object and extract the corresponding image data In this case, the plurality of image data in the time series transitions from the first image data to the second image data. If so, the motion between the first image data and the second image data is determined to be a candidate for the motion to be recognized. , and when the image data further transitions to image data other than the first and second image data, the period from the first image data to the second image data The operation of , the object is Executed The action of the recognition target and extracts the corresponding image data. The processing is executed by a computer. [Effects of the Invention]

[0007] When recognizing the motion of a target object in a plurality of frame images included in a moving image, the recognition accuracy can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an evaluation system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] FIG. 3 is an example of a flowchart showing the overall flow of evaluation processing by the evaluation system. [Figure 4] FIG. 4 is a diagram illustrating an example of the functional configuration of a registration unit realized by the information processing device in the registration phase. [Figure 5] FIG. 5 is a first diagram showing an example of a registration screen displayed on the information processing device in the registration phase and an example of action recognition data to be registered. [Figure 6] FIG. 6 is a second diagram showing an example of the registration screen displayed on the information processing device in the registration phase and the action recognition data to be registered. [Figure 7]FIG. 7 is a third diagram showing an example of the registration screen displayed on the information processing device in the registration phase and the action recognition data to be registered. [Figure 8] FIG. 8 is a diagram showing an example of a playback screen displayed on the information processing device in the registration phase. [Figure 9] FIG. 9 is a diagram for explaining an outline of the processes executed by the information processing device in the action recognition phase and the evaluation phase. [Figure 10] FIG. 10 is a diagram for explaining an outline of a posture identification function realized by the action recognition unit of the information processing device in the action recognition phase. [Figure 11] FIG. 11 is a diagram for explaining an outline of the action recognition function realized by the action recognition unit of the information processing device in the action recognition phase. [Figure 12] FIG. 12 is a diagram for explaining an outline of the selection and evaluation function realized by the evaluation unit of the information processing device in the evaluation phase. [Figure 13] FIG. 13 is a diagram illustrating an example of the functional configuration of the action recognition unit realized by the information processing device in the action recognition phase. [Figure 14] FIG. 14 is a diagram illustrating the detailed functional configuration of the posture identification unit included in the action recognition unit. [Figure 15] FIG. 15 is a diagram showing a specific example of processing by the posture identification unit included in the action recognition unit. [Figure 16] FIG. 16 is a diagram illustrating detailed functional configurations of the image specification unit for action recognition and the recognition unit included in the action recognition unit. [Figure 17] FIG. 17 is a first diagram illustrating a specific example of processing by the image-for-action-recognition specifying unit included in the action recognition unit. [Figure 18] FIG. 18 is a first diagram illustrating a specific example of processing by a recognition unit included in the action recognition unit. [Figure 19] FIG. 19 is a second diagram illustrating a specific example of processing by the image-for-action-recognition specifying unit included in the action recognition unit. [Figure 20]FIG. 20 is a second diagram illustrating a specific example of processing by the recognition unit included in the action recognition unit. [Figure 21] FIG. 21 is a third diagram illustrating a specific example of processing by the image-for-action-recognition specifying unit included in the action recognition unit. [Figure 22] FIG. 22 is a third diagram illustrating a specific example of processing by the recognition unit included in the action recognition unit. [Figure 23] FIG. 23 is a first diagram showing the advantages of the ABB type operating pattern. [Figure 24] FIG. 24 is a second diagram showing the advantages of the ABB type operating pattern. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] [First embodiment] <Evaluation system configuration> First, the overall system configuration of an evaluation system to which an information processing device according to the first embodiment is applied will be described. Fig. 1 is a diagram showing an example of the system configuration of the evaluation system. Note that, in the evaluation system to which the information processing device according to the first embodiment is applied, different processes are performed in the registration phase, and the action recognition phase and evaluation phase. Therefore, the system configuration will be described below separately for the registration phase, and the action recognition phase and evaluation phase.

[0011] (1) System configuration in the registration phase 1(a) shows an example of the system configuration of the evaluation system 100A in the registration phase. As shown in FIG. 1(a), the evaluation system 100A in the registration phase includes an imaging device 110 and an information processing device 120.

[0012] The imaging device 110 captures a moving image of an object and transmits the moving image data (an example of a plurality of image data in time series) to the information processing device 120.

[0013] An information processing program is installed in the information processing device 120. The information processing device 120 functions as a registration unit 121 by executing the program in the registration phase.

[0014] The registration unit 121 reads out the moving image data transmitted from the imaging device 110 and stored in the moving image data storage unit 123, and generates data for action recognition based on various specifications from the user 130. The registration unit 121 also stores the generated data for action recognition in the data storage unit 124 for action recognition.

[0015] The action recognition data refers to data for identifying frame images necessary for recognizing the action of a target object from among multiple frame images included in the video data captured in the action recognition phase. The action refers to a series of movements consisting of a sequence of multiple postures of the target object.

[0016] (2) System configuration in the motion recognition phase and evaluation phase 1(b) shows an example of the system configuration of the evaluation system 100B in the action recognition phase and the evaluation phase. As shown in FIG. 1(b), the evaluation system 100B in the action recognition phase and the evaluation phase includes an imaging device 140 and an information processing device 120.

[0017] The imaging device 140 captures moving images of an object and transmits the moving image data (an example of multiple pieces of image data in time series) to the information processing device 120.

[0018] The information processing device 120 functions as an action recognition unit 122 and an evaluation unit 150 in the action recognition phase and the evaluation phase.

[0019] The action recognition unit 122 reads out action recognition data from the action recognition data storage unit 124. Furthermore, the action recognition unit 122 identifies frame images necessary to recognize the action of the recognition target based on the read out action recognition data for the video image data transmitted from the imaging device 140 and acquired via the evaluation unit 150, and recognizes the action of the recognition target. Furthermore, the action recognition unit 122 cuts out frame images corresponding to the recognized action of the recognition target, and notifies the evaluation unit 150 of the cut out frame images as action data.

[0020] The evaluation unit 150 receives the moving image data transmitted by the imaging device 140 and notifies the action recognition unit 122 of the data, and also acquires action data from the action recognition unit 122 and displays it to the evaluator 160 as evaluation data.

[0021] Alternatively, when the evaluation unit 150 acquires motion data from the motion recognition unit 122, it selects a frame image to be evaluated from the acquired motion data and evaluates the posture of the target object using the selected frame image. Then, the evaluation unit 150 associates the evaluation result with the selected frame image and displays it to the evaluator 160 as evaluation data.

[0022] The function of selecting a frame image to be evaluated from the action data and evaluating the posture of the object using the selected frame image may be included in the action recognition unit 122. In this case, the action recognition unit 122 notifies the evaluation unit 150 of the evaluation result and the selected frame image, and the evaluation unit 150 associates the notified evaluation result with the selected frame image and displays them to the evaluator 160 as evaluation data.

[0023] <Hardware configuration of information processing device> Next, a description will be given of the hardware configuration of the information processing device 120. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device.

[0024] 2, the information processing device 120 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a drive device 206. The hardware components of the information processing device 120 are connected to each other via a bus 207.

[0025] The processor 201 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 201 reads various programs (for example, information processing programs, etc.) into the memory 202 and executes them.

[0026] The memory 202 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 201 and the memory 202 form a so-called computer, and the processor 201 executes various programs read onto the memory 202, causing the computer to realize various functions.

[0027] The auxiliary storage device 203 stores various programs and various data used when the various programs are executed by the processor 201. For example, the video data storage unit 123 and the action recognition data storage unit 124 are realized in the auxiliary storage device 203.

[0028] The I / F device 204 accepts operations for the information processing device 120 via the operation device 210. The I / F device 204 also outputs results of processing by the information processing device 120 and displays them via the display device 220. The communication device 205 also communicates with the imaging device 110 or the imaging device 140.

[0029] The drive device 206 is a device for loading a recording medium 230. The recording medium 230 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 230 may also include semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0030] The various programs to be installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 230 in the drive device 206 and reading the various programs recorded on the recording medium 230 by the drive device 206. Alternatively, the various programs to be installed in the auxiliary storage device 203 may be installed by being downloaded from a network (not shown) via the communication device 205.

[0031] <Evaluation process flow in the evaluation system> Next, the overall flow of the evaluation process by the evaluation systems 100A and 100B will be described. Fig. 3 is an example of a flowchart showing the overall flow of the evaluation process by the evaluation system. As shown in Fig. 3, first, the evaluation process is executed by the evaluation system 100A under the registration phase.

[0032] In step S301, the registration unit 121 of the information processing device 120 acquires video data.

[0033] In step S302, the registration unit 121 of the information processing device 120 receives various specifications for action recognition from the user 130 and generates action recognition data.

[0034] In step S303, the registration unit 121 of the information processing device 120 stores the generated action recognition data in the action recognition data storage unit .

[0035] Subsequently, the evaluation process transitions from the registration phase to the action recognition phase, and is executed by the evaluation system 100B. In step S304, the evaluation unit 150 of the information processing device 120 acquires video image data.

[0036] In step S305, the action recognition unit 122 of the information processing device 120 identifies frame images necessary for recognizing the action of the recognition target from among a plurality of frame images included in the acquired video data, based on the action recognition data.

[0037] In step S306, the action recognition unit 122 of the information processing device 120 recognizes the action of the recognition target from among the multiple frame images included in the video image data, based on the action recognition data.

[0038] In step S307, the action recognition unit 122 of the information processing device 120 extracts, as action data, frame images corresponding to the action to be recognized from the video image data.

[0039] Subsequently, the evaluation process transitions from the action recognition phase to the evaluation phase, and continues to be executed by the evaluation system 100B. In step S308, the evaluation unit 150 of the information processing device 120 evaluates the posture of the object using the extracted action data.

[0040] In step S309, the evaluation unit 150 of the information processing device 120 displays the evaluation result of the posture of the object to the evaluator 160.

[0041] <Functional configuration of the registration unit> Next, a detailed description will be given of the functional configuration of the registration unit 121 realized by the information processing device 120 in the registration phase. Fig. 4 is a diagram showing an example of the functional configuration of the registration unit realized by the information processing device in the registration phase.

[0042] As shown in FIG. 4, the registration unit 121 further includes an image data display unit 401 , a movement pattern designation unit 402 , a posture data designation unit 403 , a posture identification unit 404 , a movement recognition data registration unit 405 , and a reproduction unit 406 .

[0043] The image data display unit 401 reads out moving image data specified by the user 130 from the moving image data stored in the moving image data storage unit 123, and displays each frame image included in the read moving image data to the user 130.

[0044] The movement pattern designation unit 402 is an example of a first designation unit, and upon transition to the movement recognition phase, accepts designation of a "movement pattern" that indicates a recognition method to be used when recognizing the movement of the recognition target in the video image data. Furthermore, the movement pattern designation unit 402 accepts designation of the "movement of the recognition target" to be recognized under the designated "movement pattern."

[0045] In this embodiment, the operation patterns include the following three types (ABC type, ABB type, and BBB type). ABC type: A pattern for recognizing the target movement by specifying the target's posture when the movement begins, the target's posture while the movement is being performed, and the target's posture when the movement ends. ABB type: A pattern that recognizes the target movement by specifying the target's posture when starting the movement and the target's posture while performing the movement. BBB type: A pattern in which the target movement is recognized by specifying the posture of the target object while it is performing the movement.

[0046] The posture data designation unit 403 is an example of a second designation unit, and designates frame images necessary for recognizing the movement of the recognition target by each movement pattern as posture data from among the frame images displayed by the image data display unit 401. The posture data designation unit 403 When the movement pattern is ABC type, three types of frame images are specified as posture data, When the movement pattern is ABB type, two types of frame images are specified as posture data, If the movement pattern is BBB type, specify one type of frame image as posture data.

[0047] The posture identification unit 404 identifies the posture of the object included in each frame image displayed by the image data display unit 401. The posture identification unit 404 also assigns a marker indicating which posture indicated by the designated posture data the posture of the object included in each frame image corresponds to.

[0048] The action recognition data registration unit 405 is an example of a data registration unit, and generates action recognition data and stores it in the action recognition data storage unit 124. Specifically, the action recognition data registration unit 405 registers the following as the action recognition data: The motion pattern and the motion of the recognition target designated by the motion pattern designation unit 402, Information indicating that the posture data is specified by the posture data specifying unit 403 and the identified posture; A marker corresponding to the posture of the object included in each frame image identified by the posture identification unit 404; is stored in the action recognition data storage unit 124 in association with a plurality of frame images.

[0049] The action recognition data registration unit 405 registers the following as markers corresponding to the posture of the object: For ABC type, The marker that indicates the posture of the object when starting the action to be recognized is "A". The marker that indicates the posture of the object while performing the action to be recognized is "B", The marker that indicates the posture of the object when the recognition target action is completed is "C". "D" indicates that the attitude does not correspond to any of the attitude data. and For ABB type, The marker that indicates the posture of the object when starting the action to be recognized is "A". The marker that indicates the posture of the object while performing the action to be recognized is "B", "D" indicates that the attitude does not correspond to any of the attitude data. and For BBB type, The marker that indicates the posture of the object while performing the action to be recognized is "B", "D" indicates that the attitude does not correspond to the attitude indicated by the attitude data. It is written as follows.

[0050] The playback unit 406 plays back the video data included in the action recognition data stored in the action recognition data storage unit 124, and displays it to the user 130. When playing back the video data, the playback unit 406 changes the display mode of the seek bar depending on the type of marker added to each frame image included in the video data to be played back.

[0051] <Examples of registration screens and data for motion recognition> Next, the registration screen displayed on the information processing device 120 in the registration phase and the action recognition data to be registered will be described for each action pattern. Figures 5 to 7 are first to third diagrams showing an example of the registration screen displayed on the information processing device 120 in the registration phase and the action recognition data to be registered.

[0052] (1) Registration screen and motion recognition data for ABC type motion pattern First, the registration screen and the data for action recognition when the action pattern is ABC type will be described with reference to Fig. 5. As shown in Fig. 5, the registration screen 510 includes a motion pattern designation field 511. The example in Fig. 5 shows how "ABC type" is designated in the motion pattern designation field 511.

[0053] 5, the registration screen 510 also includes a recognition target action specification field 512. The example in FIG. 5 shows a state in which "forward bending action" is specified in the recognition target action specification field 512.

[0054] 5, the registration screen 510 also includes a display field 513 that displays each frame image included in the video data. The example in FIG. 5 shows seven frame images displayed in the display field 513.

[0055] 5, the registration screen 520 shows a state in which three types of posture data are designated from among the seven frame images displayed in the display field 513. Specifically, the registration screen 520 shows The frame image 521 with identifier = "Fr2" is specified as posture data including the posture of the object when starting the motion to be recognized (here, the forward bending motion), The frame image 522 with identifier = "Fr5" is specified as posture data including the posture of the object while performing the action to be recognized (here, the forward bending action), The frame image 523 with identifier = "Fr7" is specified as posture data including the posture of the object at the end of the motion to be recognized (here, the forward bending motion). It shows the situation.

[0056] 5, registration screen 520 shows a state in which three types of posture data have been designated, and thus "Register" button 524 can be pressed. Note that the example in FIG. 5 shows a state in which action recognition data 530 is stored in action recognition data storage unit 124 as a result of pressing "Register" button 524.

[0057] As shown in FIG. 5, the action recognition data 530 includes information items such as "frame image", "identifier", "posture data", "action to be recognized", "action pattern", and "marker".

[0058] The "frame image" includes each frame image displayed in the display field 513. The "identifier" includes an identifier that identifies each frame image displayed in the display field 513.

[0059] The "posture data" includes information indicating which of the three types of posture data each frame image displayed in display field 513 corresponds to, and the posture identified for each of the three types of posture data.

[0060] The "motion of the recognition target" includes the specified motion of the recognition target. The "motion pattern" includes the specified motion pattern.

[0061] The "marker" includes a marker assigned to each frame image displayed in display field 513. Specifically, the posture of the object included in each frame image displayed in display field 513 is identified, and if the identified posture corresponds to one of the postures identified for the three types of posture data specified, the same marker as the marker assigned to the posture data is assigned. Furthermore, if the identified posture does not correspond to any of the postures identified for the three types of posture data specified, a marker indicating that it does not correspond to any of the postures is assigned.

[0062] (2) Registration screen and motion recognition data for ABB type motion pattern Next, the registration screen and the data for action recognition when the action pattern is ABB type will be described with reference to Fig. 6. As shown in Fig. 6, the registration screen 610 includes a motion pattern specification field 611. The example in Fig. 6 shows how "ABB type" is specified in the motion pattern specification field 611.

[0063] 6, the registration screen 610 also includes a recognition target action specification field 612. The example in Fig. 6 shows how "forward bending action" is specified in the recognition target action specification field 612.

[0064] 6, the registration screen 610 also includes a display field 613 that displays each frame image included in the video data. The example in FIG. 6 shows seven frame images displayed in the display field 613.

[0065] 6, the registration screen 620 shows a state in which two types of posture data are designated from among the seven frame images displayed in the display field 613. Specifically, the registration screen 620 shows The frame image 621 with identifier = "Fr2" is specified as posture data including the posture of the object when starting the motion to be recognized (here, the forward bending motion), A frame image 622 with identifier = "Fr5" is specified as posture data including the posture of the object while performing the action to be recognized (here, the forward bending action). It shows the situation.

[0066] 6, registration screen 620 shows a state in which two types of posture data have been specified, and thus "Register" button 624 can be pressed. Note that the example in FIG. 6 shows a state in which action recognition data 630 is stored in action recognition data storage unit 124 as a result of pressing "Register" button 624.

[0067] As shown in FIG. 6, the action recognition data 630 includes information items such as "frame image", "identifier", "posture data", "action to be recognized", "action pattern", and "marker".

[0068] The "frame image" includes each frame image displayed in the display field 613. The "identifier" includes an identifier that identifies each frame image displayed in the display field 613.

[0069] The "posture data" includes information indicating which of the two types of posture data each frame image displayed in the display field 613 corresponds to, and the posture identified for each of the two types of posture data.

[0070] The "motion of the recognition target" includes the specified motion of the recognition target. The "motion pattern" includes the specified motion pattern.

[0071] The "marker" includes a marker assigned to each frame image displayed in display field 613. Specifically, the posture of the object included in each frame image displayed in display field 613 is identified, and if the identified posture corresponds to one of the postures identified for the two types of specified posture data, the same marker as the marker assigned to the posture data is assigned. Furthermore, if the identified posture does not correspond to either of the postures identified for the two types of specified posture data, a marker indicating that it does not correspond to either is assigned.

[0072] (3) Registration screen and motion recognition data for the BBB type motion pattern Next, the registration screen and the data for action recognition when the action pattern is BBB type will be described with reference to Fig. 7. As shown in Fig. 7, the registration screen 710 includes a field 711 for specifying the action pattern. The example in Fig. 7 shows how "BBB type" is specified in the field 711 for specifying the action pattern.

[0073] 7, the registration screen 710 also includes a recognition target action specification field 712. The example in Fig. 7 shows how "forward bending action" is specified in the recognition target action specification field 712.

[0074] 7, the registration screen 710 also includes a display field 713 that displays each frame image included in the video data. The example in FIG. 7 shows seven frame images displayed in the display field 713.

[0075] 7, the registration screen 720 shows a state in which one type of posture data is designated from among the seven frame images displayed in the display field 713. Specifically, the registration screen 720 A frame image 721 with identifier = "Fr5" is specified as posture data including the posture of the object while performing the action to be recognized (here, the forward bending action). It shows the situation.

[0076] 7, registration screen 720 shows a state in which one type of posture data has been designated, and thus "Register" button 724 can be pressed. Note that the example in FIG. 7 shows a state in which action recognition data 730 is stored in action recognition data storage unit 124 as a result of pressing "Register" button 724.

[0077] As shown in FIG. 7, the action recognition data 730 includes information items such as "frame image", "identifier", "posture data", "action to be recognized", "action pattern", and "marker".

[0078] The "frame image" includes each frame image displayed in the display field 713. The "identifier" includes an identifier that identifies the frame image displayed in the display field 713.

[0079] The "posture data" includes information indicating that each frame image displayed in the display field 713 is one type of specified posture data, and the posture identified for one type of posture data.

[0080] The "motion of the recognition target" includes the specified motion of the recognition target. The "motion pattern" includes the specified motion pattern.

[0081] The "marker" includes a marker assigned to each frame image displayed in display field 713. Specifically, the posture of the object included in each frame image displayed in display field 713 is identified, and if the identified posture corresponds to the posture identified for one type of specified posture data, the same marker as the marker assigned to the posture data is assigned. Furthermore, if the posture does not correspond to the posture identified for one type of specified posture data, a marker indicating that it does not correspond is assigned.

[0082] <Example of playback screen> Next, a specific example of a playback screen displayed on the information processing device 120 in the registration phase will be described. Fig. 8 is a diagram showing an example of a playback screen displayed on the information processing device in the registration phase. As described above, the playback unit 406 plays back video data included in the action recognition data stored in the action recognition data storage unit 124 and displays it to the user 130.

[0083] 8, a video playback area 801 is an area where the played video data is displayed. A seek bar 802 indicates the playback position of the video data being played. As described above, the playback unit 406 changes the display mode of the seek bar depending on the type of marker added to each frame image included in the video data being played.

[0084] In Fig. 8, a seek bar 811 indicates a seek bar when playing video data made up of frame images included in the action recognition data 530 in Fig. 5. In the example of the action recognition data 530, the "marker" includes three types of markers (A, B, and C), so the seek bar 811 has three different display modes.

[0085] 8, a seek bar 812 indicates a seek bar when playing video data made up of frame images included in the action recognition data 630 of Fig. 6. In the example of the action recognition data 630, the "marker" includes three types of markers (A, B, D), and therefore the seek bar 812 has three different display modes.

[0086] 8, a seek bar 813 indicates a seek bar when playing video data made up of frame images included in the action recognition data 730 of Fig. 7. In the example of the action recognition data 730, the "marker" includes two types of markers (B, D), so the seek bar 813 has two types of display modes.

[0087] <Outline of processing in the motion recognition phase and evaluation phase> Next, an overview of the processing executed by the information processing device 120 in the action recognition phase and evaluation phase will be described. Fig. 9 is a diagram for explaining the overview of the processing executed by the information processing device in the action recognition phase and evaluation phase. Note that, for simplicity of explanation, a case where the action pattern is ABC type is shown here.

[0088] As shown in FIG. 9, in the action recognition phase, the action recognition unit 122 of the information processing device 120 acquires moving image data and adds skeleton information to each frame image included in the moving image data (reference numeral 910).

[0089] Next, the action recognition unit 122 of the information processing device 120 identifies the posture of the object included in each frame image based on the action recognition data, thereby specifying the frame images required to recognize the action of the recognition target (reference numeral 920).

[0090] Next, the movement recognition unit 122 of the information processing device 120 recognizes the movement to be recognized (bending forward in the example of FIG. 9) based on the movement recognition data, and cuts out a frame image corresponding to the movement to be recognized as movement data (reference numeral 930).

[0091] Next, in the evaluation phase, the evaluation unit 150 of the information processing device 120 selects a frame image to be evaluated from the extracted motion data (symbol 940), and evaluates the posture of the object based on the selected frame image (symbol 950).

[0092] Next, the evaluation unit 150 of the information processing device 120 displays the evaluation result (reference numeral 960) to the evaluator 160.

[0093] <Overview of each function> Next, an overview of each function (posture identification, action recognition, selection, evaluation) of the action recognition unit 122 or evaluation unit 150 of the information processing device 120 that executes the processes shown in FIG. 9 in the action recognition phase and evaluation phase will be described.

[0094] (1) Overview of posture recognition function First, an overview of the posture identification function will be described. Fig. 10 is a diagram for explaining an overview of the posture identification function realized by the action recognition unit of the information processing device in the action recognition phase.

[0095] As shown in FIG. 10, the action recognition unit 122 of the information processing device 120 The posture of the object included in the frame image 1010, Judgment criteria (reference numerals 1021, 1022, 1023, etc.) corresponding to the posture indicated by the posture data 1020 included in the data for action recognition; By comparing these, the frame images necessary to recognize the target action are identified.

[0096] The example in Figure 10 shows the calculation of the distance (distance based on skeletal information) between the posture of the object included in the frame image 1010 and the judgment criterion (symbol 1021) corresponding to standing upright in the posture data 1020 included in the data for action recognition.

[0097] Moreover, the example in Figure 10 shows the calculation of the distance (distance based on skeletal information) between the posture of the object included in the frame image 1010 and the judgment criterion (symbol 1022) corresponding to forward bending in the posture data 1020 included in the action recognition data.

[0098] The example in Figure 10 also shows how the distance (distance based on skeletal information) is calculated between the posture of the object included in the frame image 1010 and the judgment criterion (symbol 1023) corresponding to the walking of the posture data 1020 included in the action recognition data.

[0099] As a result, the action recognition unit 122 of the information processing device 120 can identify the posture of the object included in the frame image 1010 by determining the minimum distance. As a result, the action recognition unit 122 of the information processing device 120 can specify the frame image required to recognize the motion of the recognition target. Furthermore, the action recognition unit 122 of the information processing device 120 can output a marker indicating which of the postures indicated by the posture data the posture of the object corresponds to.

[0100] The example of Fig. 10 shows that the posture of the object included in the frame image 1010 is identified as "standing upright" (reference numeral 1030). The example of Fig. 10 also shows that "A" is output as a marker (reference numeral 1040).

[0101] (2) Overview of the motion recognition function Next, an outline of the action recognition function will be described. Fig. 11 is a diagram for explaining an outline of the action recognition function realized by the action recognition unit of the information processing device in the action recognition phase.

[0102] The example in FIG. 11 shows how video data including eight frame images is acquired as video data, and the frame images necessary for recognizing the motion of the recognition target are identified.

[0103] 11 shows how the forward bending motion is recognized based on the identified frame images, and the corresponding frame images (frame images with identifiers Fr1 to Fr7) are extracted as motion data. -Frame image with marker "A" attached, -Frame image with marker "B" attached, -Frame image with marker "C" attached, During this period, the object is recognized as having performed the action to be recognized, and the frame image is extracted as action data.

[0104] Although not shown in Figure 11, in the case of the ABB type, A frame image (an example of first image data) to which a marker "A" is attached, A frame image with a marker "B" (an example of second image data), During this period, the object is recognized as having performed the action to be recognized, and the frame image is extracted as action data.

[0105] In addition, in the case of BBB type, -Frame image with marker "B" attached, During this period, the object is recognized as having performed the action to be recognized, and the frame image is extracted as action data.

[0106] (3) Overview of the selection and evaluation function Next, an overview of the selection and evaluation function will be described. Fig. 12 is a diagram for explaining the overview of the selection and evaluation function realized by the evaluation unit of the information processing device in the evaluation phase.

[0107] In Fig. 12, reference numeral 1210 indicates that frame images corresponding to a forward bending movement are extracted as movement data, and a selection index (in the example of Fig. 12, "waist angle") for selecting a frame image to be evaluated is calculated for each extracted frame image. Note that, due to space limitations, reference numeral 1210 indicates only a portion of the extracted movement data in Fig. 11.

[0108] In addition, in FIG. 12, reference numeral 1220 indicates that the frame image with identifier = "Fr4" has been selected as the frame image including the object with the smallest selection indicator = "waist angle".

[0109] Also, in Figure 12, reference numeral 1230 indicates the state in which evaluation indices (in the example of Figure 12, "waist angle," "elbow angle," and "knee angle") for evaluating the posture of the object included in the selected frame image are calculated.

[0110] <Functional configuration of the action recognition unit> Next, a detailed description will be given of the functional configuration of the action recognition unit 122 realized by the information processing device 120 in the action recognition phase. Fig. 13 is a diagram showing an example of the functional configuration of the action recognition unit realized by the information processing device in the action recognition phase.

[0111] As shown in FIG. 13, the action recognition unit 122 includes a moving image data acquisition unit 1301 , a posture identification unit 1302 , an image specification unit for action recognition 1303 , and a recognition unit 1304 .

[0112] The video data acquisition unit 1301 acquires video data, adds skeletal information to each frame image included in the acquired video data, and sequentially notifies the posture identification unit 1302 of the frame images to which skeletal information has been added.

[0113] The posture identification unit 1302 reads posture data included in the action recognition data stored in the action recognition data storage unit 124. The posture identification unit 1302 also - Judgment criteria corresponding to the posture indicated by the read posture data, The posture of the object included in the frame images sequentially notified by the video data acquisition unit 1301, and and calculates the distance between them. As a result, the posture identification unit 1302 determines which posture, among those indicated by the read posture data, corresponds to the determination criterion with the smallest distance, and identifies the posture of the object contained in the frame images notified sequentially. Furthermore, by identifying the posture of the object contained in the frame images notified sequentially, the posture identification unit 1302 identifies frame images necessary for recognizing the action of the recognition target from the frame images notified sequentially by the video data acquisition unit 1301. Furthermore, the posture identification unit 1302 adds a marker corresponding to the identified posture of the object to the frame image, and notifies the action recognition image identification unit 1303 sequentially.

[0114] The image specification unit for action recognition 1303 removes frame images that become noise from among the frame images sequentially notified by the posture identification unit 1302. In this embodiment, a "frame image that becomes noise" refers to a frame image in which the same type of marker does not appear consecutively for a predetermined number of frames or more.

[0115] Furthermore, the action recognition image specifying unit 1303 sequentially notifies the recognition unit 1304 of the frame images after noise removal, from which the frame images that become noise have been removed.

[0116] The recognition unit 1304 reads out a movement pattern included in the movement recognition data stored in the movement recognition data storage unit 124. The recognition unit 1304 also determines whether or not the noise-removed frame images sequentially notified by the movement recognition image specification unit 1303 transition in accordance with the read movement pattern.

[0117] If the recognition unit 1304 determines that the noise-removed frame images sequentially notified by the action recognition image specification unit 1303 transition in accordance with the read-out action pattern, it cuts out the frame images that transition in accordance with the read-out action pattern. Furthermore, the recognition unit 1304 notifies the evaluation unit 150 of the cut-out frame images as action data.

[0118] On the other hand, if the recognition unit 1304 determines that the noise-removed frame images sequentially notified by the action recognition image specification unit 1303 do not transition in accordance with the read-out action pattern, it does not cut out the frame image.

[0119] <Details of the functional configuration of the posture identification unit included in the action recognition unit> Next, a detailed description will be given of the functional configuration of the posture identification unit 1302 included in the action recognition unit 122. Fig. 14 is a diagram showing the detailed functional configuration of the posture identification unit included in the action recognition unit. As shown in Fig. 14, the posture identification unit 1302 has an identified posture acquisition unit 1401 and a determination unit 1402.

[0120] The identification posture acquisition unit 1401 reads out the action recognition data stored in the action recognition data storage unit 124 in the registration phase, and notifies the determination unit 1402. Specifically, the identification posture acquisition unit 1401 reads out the posture data and markers from the action recognition data, and notifies the determination unit 1402.

[0121] In FIG. 14, reference numeral 1410 denotes · Action to be recognized = "forward bending action", -Movement pattern = "ABC type", are the posture data and markers for - The posture of the object when starting the action to be recognized = "upright", marker = "A", Posture of the object during the execution of the action to be recognized = "bending forward", marker = "B", - Posture of the object when the action to be recognized ends = "walking", marker = "C", Includes:

[0122] The determination unit 1402 acquires each frame image included in the moving image data acquired by the moving image data acquisition unit 1301, and each frame image to which skeletal information has been added by the moving image data acquisition unit 1301. The determination unit 1402 also identifies which of the postures (reference numeral 1410) indicated by the posture data notified by the identified posture acquisition unit 1401 corresponds to the posture of the object included in each acquired frame image, and outputs the posture identification result.

[0123] Specifically, the determination unit 1402 reads out, from the reference posture data storage unit 1403, a determination criterion corresponding to the posture (reference numeral 1410) indicated by the posture data notified by the identification posture acquisition unit 1401. The determination unit 1402 then calculates the distance between the read determination criterion and the posture of the object included in each frame image, and determines the smallest distance, thereby identifying which posture (reference numeral 1410) indicated by the posture data notified by the identification posture acquisition unit 1401 corresponds. The reference posture data storage unit 1403 stores determination criteria for identifying each posture of the object.

[0124] In FIG. 14, the reference numeral 1420 denotes - Criteria for identifying whether the object's posture is "upright"; - Criteria for identifying whether the object's posture is "bent forward" - Criteria for identifying whether the object's posture is "walking" or not, is read out.

[0125] When the determination unit 1402 identifies the posture of the object included in each frame image as one of the postures (reference numeral 1410) indicated by the posture data notified by the identified posture acquisition unit 1401, the determination unit 1402 assigns a marker corresponding to the identified posture to the corresponding frame image. In the example of reference numeral 1410, the determination unit 1402 assigns one of "A", "B", or "C" to the corresponding frame image.

[0126] On the other hand, if the posture of the object included in each frame image is determined not to correspond to any of the postures indicated by the posture data notified by the identified posture acquisition unit 1401, the determination unit 1402 assigns a marker indicating that the posture does not correspond to any of the postures to the corresponding frame image. The determination unit 1402 assigns "D" to the corresponding frame image.

[0127] The determination unit 1402 notifies the action recognition image specification unit 1303 of each frame image to which a marker has been added as a posture identification result.

[0128] <Specific example of processing by the posture identification unit included in the action recognition unit> Next, a specific example of the processing by the posture identification unit 1302 included in the action recognition unit 122 will be described. Fig. 15 is a diagram showing a specific example of the processing by the posture identification unit included in the action recognition unit.

[0129] In FIG. 15, a frame image 1510 is a frame image acquired by the determination unit 1402 of the posture identification unit 1302, and is a frame image to which skeleton information has been added.

[0130] 15, reference numerals 1421 to 1423 indicate details of the determination criteria corresponding to each posture indicated by the posture data. As indicated by reference numerals 1421 to 1423, the determination criteria corresponding to each posture indicated by the posture data include, as information items, the body part used to determine the minimum distance and the physical quantity used to determine the minimum distance. The example of FIG. 15 indicates that the determination criteria for each posture include "armpit angle," "waist angle," "knee angle," etc.

[0131] The determination unit 1402 of the posture identification unit 1302 calculates the distance (Euclidean distance) between the skeleton information assigned to the frame image 1510 and the determination criterion corresponding to each posture indicated by the posture data. Specifically, the determination unit 1402 of the posture identification unit 1302 calculates the Euclidean distance D p Calculate.

[0132]

number

[0133] The determination unit 1402 of the posture identification unit 1302 calculates the Euclidean distance D p Calculate the minimum Euclidean distance D p The posture is identified by determining the posture (see reference numeral 1520). Furthermore, the determining unit 1402 of the posture identifying unit 1302 assigns a marker corresponding to the identified posture to the frame image 1510, and outputs it as a posture identification result (see reference numeral 1530).

[0134] In the example of FIG. 15, the determination unit 1402 of the posture identification unit 1302 performs the following: The Euclidean distance D calculated between the posture of the object included in the frame image 1510 and the “upright” criterion (reference numeral 1421) p, The Euclidean distance D calculated between the posture of the object included in the frame image 1510 and the “forward bending” criterion (reference numeral 1422) p , The Euclidean distance D calculated between the posture of the object included in the frame image 1510 and the “walking” criterion (reference numeral 1423) p , Among them, the minimum Euclidean distance D p The determination unit 1402 of the posture identification unit 1302 determines the Euclidean distance D p are equal to or greater than a predetermined threshold, it is determined that the posture of the object included in frame image 1510 does not correspond to any of the postures indicated by the posture data. In this case, determination unit 1402 of posture identification unit 1302 assigns a marker to frame image 1510 indicating that it does not correspond to any of the postures indicated by the posture data, and outputs it as the posture identification result.

[0135] <Functional configuration of the action recognition image specifying unit and recognition unit included in the action recognition unit> Next, a detailed description will be given of the functional configuration of the image specification unit for action recognition 1303 and the recognition unit 1304 included in the action recognition unit 122. Fig. 16 is a diagram showing the detailed functional configuration of the image specification unit for action recognition and the recognition unit included in the action recognition unit. As shown in Fig. 16, the image specification unit for action recognition 1303 further includes a continuity determination unit 1601 and a noise removal unit 1602, and the recognition unit 1304 further includes an order determination unit 1611 and a final state determination unit 1612.

[0136] The continuity determination unit 1601 sequentially acquires the frame images with markers added that are output from the determination unit 1402, and notifies the noise removal unit 1602. At this time, the continuity determination unit 1601 determines whether or not the same type of marker has been added to each frame image for a predetermined number of consecutive frames or more. Furthermore, if the continuity determination unit 1601 determines that the same type of marker has not been added to the corresponding frame image for a predetermined number of consecutive frames or more, it notifies the noise removal unit 1602 that the same type of marker has not been added to the corresponding frame image for a predetermined number of consecutive frames or more.

[0137] The noise removal unit 1602 removes frame images that have not been notified by the continuity determination unit 1601 as having the same type of marker added consecutively for a predetermined number of frames or more. The noise removal unit 1602 also notifies the order determination unit 1611 of the recognition unit 1304 of the frame images that have not been removed.

[0138] The order determining unit 1611 reads out the action recognition data from the action recognition data storage unit 124, and determines whether the frame images notified in sequence by the noise removing unit 1602 correspond to a predetermined order.

[0139] Specifically, the order determination unit 1611 reads out the movement pattern from the movement recognition data and determines whether the frame images notified in sequence by the noise removal unit 1602 correspond to the order specified by the movement pattern. The order determination unit 1611 also determines the frame images determined to correspond to the order specified by the movement pattern as candidates for movement data.

[0140] In FIG. 16, the reference numeral 1620 denotes · Action to be recognized = "forward bending action", -Movement pattern = "ABC type", This indicates that if it is determined that the markers added to the frame images notified sequentially by the noise removal unit 1602 correspond to the order A → B → C, the corresponding frame image is determined to be a candidate for movement data of the “forward bending movement.”

[0141] The end state determination unit 1612 determines whether a frame image with a predetermined marker is notified after the frame image determined to be a candidate for motion data, among the frame images notified by the order determination unit 1611. The predetermined marker here refers to a marker other than the marker corresponding to the posture of the object when the motion to be recognized is completed. Specifically, When the operation pattern is "ABC type", the specified marker is a marker other than "C" (i.e., "A", "B", "D"), When the operation pattern is "ABB type", the specified marker is a marker other than "B" (i.e., "A", "D"), When the operation pattern is "BBB type", the specified marker is a marker other than "B" (i.e., "D"). Refers to...

[0142] If the end state determination unit 1612 determines that a frame image with a predetermined marker has been notified after a frame image determined to be a candidate for motion data, the end state determination unit 1612 recognizes the candidate for motion data as the motion of the recognition target. The end state determination unit 1612 also cuts out the frame image corresponding to the recognized motion of the recognition target as motion data, transmits it to the evaluation unit 150, and then proceeds to the next frame image.

[0143] On the other hand, if it is determined that a frame image with a specified marker has not been notified after the frame image determined to be a candidate for motion data, the end state determination unit 1612 proceeds to the next frame image without extracting the motion data.

[0144] <Specific examples of the image specification unit for action recognition included in the action recognition unit and the processing by the action recognition unit> Next, a specific example of processing by the image specification unit for action recognition 1303 and the recognition unit 1304 included in the action recognition unit 122 will be described. Note that, below, specific examples of processing by the image specification unit for action recognition 1303 and the recognition unit 1304 will be described for each of the action patterns ("ABC type", "ABB type", and "BBB type").

[0145] (1) Operation pattern = "ABC type" First, a specific example of processing by the image specification unit for action recognition 1303 and the recognition unit 1304 when the action pattern is "ABC type" will be described. Fig. 17 is a first diagram showing a specific example of processing by the image specification unit for action recognition included in the action recognition unit.

[0146] 17(a), (b), and (c) show frame images (frame images with markers added) sequentially input to the image specifying unit for action recognition 1303 on the left side, and frame images sequentially output from the image specifying unit for action recognition 1303 on the right side. When the action pattern is "ABC type," as shown in FIG. 17(a), (b), and (c), a marker of "A," "B," "C," or "D" is added to the frame image.

[0147] The examples of Figures 17(a), (b), and (c) show how the action recognition image specifying unit 1303 removes corresponding frame images when the same type of marker has not been added for two or more consecutive frames. Specifically, the example of Figure 17(a) shows how the eighth frame image (frame image with marker="A") has been removed after it has been determined that the same type of marker has not been added for two or more consecutive frames. Also, the example of Figure 17(a) shows how the ninth frame image (frame image with marker="C") has been removed after it has been determined that the same type of marker has not been added for two or more consecutive frames.

[0148] Moreover, the example in FIG. 17(b) shows that no frame images were removed because there were no frame images to which the same type of marker was not added for two or more consecutive frames.

[0149] In addition, the example in Figure 17(c) shows that the fifth frame image (the frame image with marker="D") was removed because it was determined that the same type of marker had not been added for two or more consecutive frames.

[0150] Fig. 18 is a first diagram showing a specific example of processing by the recognition unit included in the action recognition unit. The left sides of Fig. 18(a), (b), and (c) show frame images output from the action recognition image specification unit 1303, and are the same as the right sides of Fig. 17(a), (b), and (c).

[0151] The example in Figure 18(a) shows that the frame images output by the image specification unit for action recognition 1303 were not recognized as a "forward bending action" because they were determined not to correspond to the order A → B → C.

[0152] The example in Fig. 18(b) shows that the frame images output by the image specification unit for action recognition 1303 are determined to correspond to the order A → B → C, and are determined to be candidates for action data. On the other hand, the example in Fig. 18(b) shows that the frame image with the "C" marker was not recognized as a "forward bending action" because no frame image with a marker other than "C" was notified after the frame image with the "C" marker.

[0153] The example in Fig. 18(c) shows that the frame images output by the image specification unit for action recognition 1303 are determined to correspond to the order A → B → C, and are determined to be candidates for action data. Furthermore, the example in Fig. 18(c) shows that a frame image with a marker other than "C" is notified after a frame image with a marker "C", and therefore is recognized as a "forward bending action."

[0154] (2) Operation pattern = "ABB type" Next, a specific example of processing by the image specification unit for action recognition 1303 and the recognition unit 1304 when the action pattern is "ABB type" will be described. Fig. 19 is a second diagram showing a specific example of processing by the image specification unit for action recognition included in the action recognition unit.

[0155] 19(a), (b), and (c) show frame images (frame images with markers added) sequentially input to the image specifying unit for action recognition 1303 on the left side, and frame images sequentially output from the image specifying unit for action recognition 1303 on the right side. When the action pattern is "ABB type," as shown in FIG. 19(a), (b), and (c), a marker of "A," "B," or "D" is added to the frame image.

[0156] 19(a), (b), and (c) show how the action recognition image specifying unit 1303 removes corresponding frame images when the same type of marker has not been added for two or more consecutive frames. Specifically, the example in Fig. 19(a) shows how the eighth frame image (a frame image with marker="A") has been removed after it was determined that the same type of marker has not been added for two or more consecutive frames.

[0157] Moreover, the example in FIG. 19(b) shows that no frame images were removed because there were no frame images to which the same type of marker was not added for two or more consecutive frames.

[0158] In addition, the example in Figure 19(c) shows that the fifth frame image (the frame image with marker="D") was removed because it was determined that the same type of marker had not been added for two or more consecutive frames.

[0159] Fig. 20 is a second diagram showing a specific example of processing by the recognition unit included in the action recognition unit. The left sides of Fig. 20(a), (b), and (c) show frame images output from the action recognition image specification unit 1303, and are the same as the right sides of Fig. 19(a), (b), and (c).

[0160] The example in Figure 20(a) shows that the frame images output by the image specification unit for action recognition 1303 were not recognized as a "forward bending action" because they were determined not to correspond to the order A → B.

[0161] The example in Fig. 20(b) shows how the frame images output by the image specification unit for action recognition 1303 are determined to correspond to the order A → B and are determined to be candidates for action data. Furthermore, the example in Fig. 20(b) shows that a frame image with a marker other than "B" is notified after a frame image with a marker "B", and therefore is recognized as a "forward bending action."

[0162] Similarly, the example in Fig. 20(c) shows how the frame images output by the image specification unit for action recognition 1303 are determined to correspond to the order A → B and are determined to be candidates for action data. Furthermore, the example in Fig. 20(c) shows that a frame image with a marker other than "B" is notified after a frame image with a marker "B", and therefore is recognized as a "forward bending action."

[0163] (3) Operation pattern = "BBB type" Next, a specific example of processing by the image specification unit for action recognition 1303 and the recognition unit 1304 when the action pattern is "BBB type" will be described. Fig. 21 is a third diagram showing a specific example of processing by the image specification unit for action recognition included in the action recognition unit.

[0164] 21(a), (b), and (c) show frame images (frame images with markers added) sequentially input to the image specifying unit for action recognition 1303 on the left side, and frame images sequentially output from the image specifying unit for action recognition 1303 on the right side. When the movement pattern is "BBB type," as shown in FIG. 21(a), (b), and (c), the frame images are added with either a "B" or a "D" marker.

[0165] 21(a), (b), and (c) show how the action recognition image specifying unit 1303 removes corresponding frame images when the same type of marker is not added for two or more consecutive frames. Specifically, the example in Fig. 21(a) shows how none of the frame images were removed because there were no frame images to which the same type of marker was not added for two or more consecutive frames.

[0166] Similarly, the example in FIG. 21(b) shows that no frame images were removed because there were no frame images to which the same type of marker was not added for two or more consecutive frames.

[0167] On the other hand, the example in Figure 21(c) shows that the fifth frame image (the frame image with marker="D") was removed because it was determined that the same type of marker had not been added for two or more consecutive frames.

[0168] Fig. 22 is a third diagram showing a specific example of processing by the recognition unit included in the action recognition unit. The left sides of Fig. 22(a), (b), and (c) show frame images output from the action recognition image specification unit 1303, and are the same as the right sides of Fig. 21(a), (b), and (c).

[0169] The example in Fig. 22(a) shows how each frame image output from the image-for-motion-recognition specifying unit 1303 is determined to contain a "B" marker and is determined to be a candidate for motion data. Furthermore, the example in Fig. 22(a) shows how a frame image with a marker other than "B" is notified after a frame image with a "B" marker, and therefore is recognized as a "forward bending image."

[0170] Similarly, the example in FIG. 22(b) shows that among the frame images output from the motion recognition image identification unit 1303, it is determined that the marker "B" is included, and it is determined as a candidate for motion data. Further, the example in FIG. 22(b) shows that after the frame image with the marker "B" is given, a frame image with a marker other than "B" is notified, so it shows the state of being recognized as a "forward flexion image".

[0171] Similarly, the example in FIG. 22(c) shows that among the frame images output from the motion recognition image identification unit 1303, it is determined that the marker "B" is included, and it is determined as a candidate for motion data. Further, the example in FIG. 22(c) shows that after the frame image with the marker "B" is given, a frame image with a marker other than "B" is notified, so it shows the state of being recognized as a "forward flexion image".

[0172] <Advantages of the ABB type motion pattern> Next, the advantages of the ABB type motion pattern will be described. Here, for comparison, cases where it is better to apply the "ABB type" than to apply the "BBB type", and cases where it is better to apply the "ABB type" than to apply the "ABC type" will be described respectively.

[0173] (1) Case where it is better to apply the "ABB type" than the "BBB type" <(0000712)>FIG. 23 is a first diagram showing the advantages of the ABB type motion pattern. In FIG. 23, reference numeral 2310 indicates each frame image of the moving image data captured when the operator performing the forward flexion motion takes an appropriate forward flexion posture. As shown by reference numeral 2310, when the operator takes an appropriate forward flexion posture, by applying the motion pattern = "BBB type", the motion recognition unit 122 can recognize the forward flexion motion.

[0174] 23, reference numeral 2320 denotes each frame image of video data captured when the performer performing the forward bending movement is, for example, an elderly person and is unable to assume an appropriate forward bending posture (unable to bend forward deeply). As shown by reference numeral 2320, if the performer is unable to assume an appropriate forward bending posture, the movement recognition unit 122 cannot recognize the forward bending movement even if the movement pattern = "BBB type" is applied.

[0175] Therefore, it is assumed that the user 130 relaxes the criteria for determining posture="bending forward" to make it easier to identify posture="bending forward." In this case, as shown by the reference numeral 2321, even if the performer performing the bending forward motion is unable to assume an appropriate bending forward posture, the motion recognition unit 122 will be able to identify it as "bending forward."

[0176] On the other hand, in this case, when the performer performing the bending forward action adopts an appropriate bending forward posture, the action recognition unit 122 is more likely to erroneously classify the posture, as shown by reference numeral 2311. Reference numeral 2311 shows how, in the first, second, sixth, and seventh frame images, the posture is erroneously classified as "bending forward" when it should have been classified as "standing upright."

[0177] In this situation, suppose that the user 130 changes the movement pattern from "BBB type" to "ABB type." In this case, the movement recognition unit 122 can recognize the "bending forward movement" regardless of whether the performer is able to take an appropriate bending forward posture or is unable to take an appropriate bending forward posture.

[0178] Specifically, in the case of the "BBB type," the action recognition unit 122 erroneously classified the first, second, sixth, and seventh frame images as "bending forward" when they should have been classified as something other than "bending forward," as shown by reference numeral 2311 in Fig. 23. In contrast, in the case of the "ABB type," the action recognition unit 122 can classify the frame images as "standing upright," as shown by reference numeral 2312 in Fig. 23.

[0179] Similarly, in the case of the "BBB type," the action recognition unit 122 erroneously classified the first, second, fifth, and sixth frame images as "bending forward" when they should have been classified as something other than "bending forward," as shown by reference numeral 2321 in Fig. 23. In contrast, in the case of the "ABB type," the action recognition unit 122 can classify the frame images as "standing upright," as shown by reference numeral 2322 in Fig. 23.

[0180] In this way, by changing the movement pattern from "BBB type" to "ABB type," the movement recognition unit 122 can recognize the "bending forward movement" whether the performer is able to take an appropriate bending forward posture or is unable to take an appropriate bending forward posture.

[0181] (2) Cases where the "ABB type" is better than the "ABC type" Figure 24 is a second diagram illustrating the advantages of the ABB type movement pattern. In Figure 24, reference numeral 2410 indicates each frame image of video data captured when an actor performing a forward bending movement assumes an appropriate posture. The appropriate posture here refers to the appropriate posture for the "ABC type" movement pattern (i.e., the actor's posture at the start, during the movement, and at the end is appropriate).

[0182] In this case, as shown by reference numeral 2411, when the movement pattern = "ABC type" is applied, the movement recognition unit 122 can identify the posture of the performer at the start, the posture of the performer while performing, and the posture of the performer at the end. Then, the movement recognition unit 122 can appropriately assign markers "A", "B", and "C". As a result, the movement recognition unit 122 can recognize the "forward bending movement."

[0183] Furthermore, as shown by reference numeral 2412, even when the movement pattern = "ABB type" is applied, the movement recognition unit 122 can identify the posture of the performer at the start and during the movement, and appropriately assign markers "A," "B," and "D." As a result, the movement recognition unit 122 can recognize the "forward bending movement."

[0184] On the other hand, in Fig. 24, reference numeral 2420 indicates each frame image of video data captured when an actor performing a forward bending movement was unable to assume an appropriate posture. In this case, "unable to assume an appropriate posture" refers to an actor being unable to assume an appropriate posture in the case of the movement pattern = "ABC type." Specifically, this refers to an actor being able to assume an appropriate posture at the start and during the movement, but being unable to assume an appropriate posture at the end.

[0185] In this case, as shown by reference numeral 2421, when the movement pattern = "ABC type" is applied, the movement recognition unit 122 can identify the posture of the performer at the start and during the execution, but cannot identify the posture of the performer at the end. Then, the movement recognition unit 122 assigns markers "A", "B", and "D". As a result, the movement recognition unit 122 cannot recognize the "forward bending movement".

[0186] In contrast to this, as shown by reference numeral 2422, when the movement pattern = "ABB type" is applied, the movement recognition unit 122 can identify the posture of the performer at the start and during the execution, and appropriately assign markers "A", "B", and "D". As a result, the movement recognition unit 122 can recognize the "forward bending movement".

[0187] In this way, by applying the "ABB type" as the movement pattern, the movement recognition unit 122 can recognize the "forward bending movement" even if the performer is unable to assume an appropriate posture at the end.

[0188] As is clear from the above description, the action recognition unit 122 of the information processing device 120 according to the first embodiment includes a posture identification unit 1302 and a recognition unit 1304. The posture identification unit 1302 From among the multiple frame images included in the video data, first and second frame images are identified that respectively include, as the posture of the performer, the posture that the performer should take when starting the movement to be recognized and the posture that the performer should take while performing the movement to be recognized.

[0189] Furthermore, when recognizing the motion of the recognition target using the ABB type motion pattern, the recognition unit 1304: A plurality of frame images included in the video data transition from a first frame image to a second frame image, Furthermore, when a transition occurs to a frame image other than the first and second frame images (other than the first and second image data), It is recognized that the performer performed the action to be recognized from the first frame image to the second frame image.

[0190] In this way, by recognizing the movement of the object to be recognized based on the ABB type movement pattern, according to the first embodiment, it is possible to improve the recognition accuracy when recognizing the movement of the object to be recognized in multiple frame images included in a moving image.

[0191] [Second embodiment] In the first embodiment, the "bending forward motion" is used as an example of the motion to be recognized, but the motion to be recognized is not limited to the "bending forward motion." In addition, any posture can be specified for the posture of the object when starting the motion to be recognized and the posture of the object when finishing the motion to be recognized.

[0192] Furthermore, in the first embodiment, when removing noise, the condition is that the same type of marker appears in two or more consecutive frames, but the number of consecutive frames is not limited to two.

[0193] Furthermore, in the first embodiment, the case where the same information processing device 120 is used in the registration phase and the action recognition phase has been described, but separate information processing devices may be used for the registration phase and the action recognition phase.

[0194] Furthermore, in the above first embodiment, the evaluation unit 150 and the action recognition unit 122 in the information processing device 120 are described as separate functions, but the evaluation unit 150 and the action recognition unit 122 may be realized as a single function.

[0195] The present invention is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0196] 100A, 100B: Evaluation system 110: Imaging device 111: Layered encoding device 120: Information processing device 121: Registration Department 122: Motion recognition section 133: Video display section 140: Imaging device 150: Evaluation section 401: Image data display unit 402: Operation pattern specification section 403: Attitude data specification section 404: Posture recognition unit 405: Action recognition data registration unit 406: Playback section 510, 520: Registration screen 530: Data for motion recognition 610, 620: Registration screen 630: Data for motion recognition 710, 720: Registration screen 730: Data for motion recognition 800: Playback screen 811~813: Seek bar 1301: Video data acquisition unit 1302: Posture recognition unit 1303: Image identification unit for motion recognition 1304 :Recognition part 1401: Identification posture acquisition unit 1402: Judgment section 1601: Continuity determination unit 1602: Noise removal section 1611 :Order judgment part 1612: End status determination section

Claims

1. Identifying, from among the plurality of time-series image data, as the posture of the object, first image data including the posture that the object should take when starting the motion of the recognition target, and second image data including the posture that the object should take while performing the motion of the recognition target; When recognizing the movement of the recognition target based on two types of postures of the object and extracting corresponding image data, if the plurality of image data in the time series transitions from the first image data to the second image data, the movement between the first image data and the second image data is determined to be a candidate for the movement of the recognition target, and if the plurality of image data in the time series transitions to image data other than the first and second image data, the movement between the first image data and the second image data is recognized as the movement of the recognition target performed by the object, and the corresponding image data is extracted. An information processing program that causes a computer to execute a process.

2. causing the computer to execute a process of storing, in a storage unit, information regarding a posture that the object should take when starting the motion of the recognition target, and information regarding a posture that the object should take while performing the motion of the recognition target; The information processing program according to claim 1 , wherein the first image data and the second image data are identified by referring to the storage unit.

3. 2. The information processing program according to claim 1, wherein it is determined that a transition has occurred to image data other than the first and second image data when image data other than the first and second image data continues for a predetermined number of frames or more.

4. When image data including a posture that the object should take when starting the movement of the recognition target continues for a predetermined number of frames or more, the continuous image data is identified as the first image data; When image data including a posture that the object should take while performing the action to be recognized continues for a predetermined number of frames or more, the continuous image data is identified as the second image data. The information processing program according to claim 3 .

5. A plurality of time-series image data are arranged in time sequence and displayed. Accepting a designation of a motion pattern for recognizing a motion of a recognition target performed by an object; Accepting designation of image data corresponding to the movement pattern from among the plurality of image data displayed in a time series arrangement; Identifying a posture of the object included in the image data for which the designation has been accepted, and displaying the identified posture at a position corresponding to a position at which the image data for which the designation has been accepted is displayed; The movement pattern for which the designation has been accepted, the identified posture, and the image data for which the designation has been accepted are stored in a storage unit in association with each other. An information processing program that causes a computer to execute a process.

6. When the user specifies the movement pattern for recognizing the movement of the object based on two different postures of the object, Designation of image data including the posture that the object should take when starting the action to be recognized; Specifying image data including a posture that the object should assume while performing the action to be recognized; The information processing program according to claim 5, wherein the program accepts the following:

7. When the user specifies the movement pattern for recognizing the movement of the object based on the three types of postures of the object, Designation of image data including the posture that the object should take when starting the action to be recognized; Specifying image data including a posture that the object should assume while performing the action to be recognized; Designation of image data including the posture that the object should assume when completing the action to be recognized; The information processing program according to claim 5, wherein the program accepts the following:

8. When a motion pattern is specified for recognizing the motion of a recognition target based on one type of posture of the target object, Specifying image data containing the posture that the object should assume while performing the action to be recognized The information processing program according to claim 5, wherein the program accepts the following:

9. 9. An information processing program according to claim 6, wherein information indicating which of the postures of the object included in the displayed plurality of image data corresponds to which of the postures of the object included in the image data for which the designation has been accepted is stored in the storage unit in association with the displayed plurality of image data.

10. 10. The information processing program according to claim 9, wherein when the plurality of image data are played back, a display mode of each position of a seek bar is displayed in a display mode corresponding to the information stored in the storage unit.

11. A process of identifying, from among a plurality of time-series image data, first image data including a posture that the object should take when starting the motion of the recognition target, and second image data including a posture that the object should take while performing the motion of the recognition target; When recognizing a movement of the recognition target based on two types of postures of the target and extracting corresponding image data, if the plurality of time-series image data transitions from the first image data to the second image data, the movement between the first image data and the second image data is determined to be a candidate for the movement of the recognition target, and if the plurality of time-series image data transitions to image data other than the first and second image data, the movement between the first image data and the second image data is recognized as the movement of the recognition target performed by the target, and the corresponding image data is extracted. An information processing method performed by a computer.

12. a posture identification unit that identifies, from a plurality of time-series image data, first image data including a posture that the object should take when starting a motion to be recognized, and second image data including a posture that the object should take while performing the motion to be recognized, as postures of the object; a recognition unit that, when recognizing a movement of the recognition target based on two types of postures of the target and cutting out corresponding image data, determines, when the plurality of time-series image data transitions from the first image data to the second image data, a movement between the first image data and the second image data as a candidate for the movement of the recognition target, and when the plurality of time-series image data transitions to image data other than the first and second image data, recognizes a movement between the first image data and the second image data as the movement of the recognition target performed by the target, and cuts out corresponding image data; An information processing device having the above.

13. A process of arranging and displaying a plurality of time-series image data in a time-series manner; A process of receiving a designation of a motion pattern for recognizing a motion of a recognition target performed by an object; a process of accepting designation of image data corresponding to the movement pattern from among the plurality of image data arranged in time series and displayed; a process of identifying a posture of the object included in the image data for which the designation has been accepted, and displaying the identified posture at a position corresponding to a position at which the image data for which the designation has been accepted is displayed; a process of storing the specified movement pattern, the identified posture, and the specified image data in a storage unit in association with each other; An information processing method performed by a computer.

14. a display unit that displays a plurality of time-series image data in a time-series arrangement; a first designation unit that accepts designation of a motion pattern for recognizing a motion of a recognition target performed by an object; a second designation unit that accepts designation of image data corresponding to the movement pattern from among the plurality of image data displayed in a time series arrangement; a posture identifying unit that identifies a posture of the object included in the image data whose specification has been accepted, and displays the identified posture at a position corresponding to a position where the image data whose specification has been accepted is displayed; a data registration unit that associates the designated movement pattern, the identified posture, and the designated image data with each other and stores them in a storage unit; An information processing device having the above.

Images