Motion identification program, information processing device, and motion identification method

Abstract

To provide an action specifying program which can quantitatively represent action of a care person, an information processing apparatus, and an action specifying method.SOLUTION: An action specifying program includes: specifying first image data and second image data from video data; calculating first optical flows indicating displacement for each pixel in the first image data and the second image data; sorts each of the first optical flows for each pixel into a plurality of groups on the basis of a direction corresponding to each of the first optical flows for each pixel; specifying, for every multiple groups, action characteristics of an object included in the first image data and the second image data on the basis of a length corresponding to each of the first optical flows for every multiple pixels included in each group; and outputting the specified action characteristics.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an action identification program, an information processing device, and an action identification method, and more particularly, to an action identification program, an information processing device, and an action identification method that use video data of an object captured to extract characteristics of the action of the object captured in the video data and recognize the action from the extracted characteristics. [Background technology]

[0002] For example, in the field of nursing care, nursing care costs are calculated according to the work that a caregiver does for a care recipient. Specifically, such nursing care cost calculations may be performed, for example, by qualitative human judgment.

[0003] However, for example, if the criteria for judgment differ for each caregiver, the calculation of nursing care costs as described above may not always be performed accurately. For this reason, in recent years, various studies have been conducted to quantitatively evaluate the content of the work performed by the caregiver (the actions of the caregiver). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-021944 A [Non-patent literature]

[0005] [Non-Patent Document 1] Yuki Niitsuma, Syunpei Torii, Yuichi Yaguchi, Ryuichi Oka: “Time-segmentation and position-free recognition of air-drawn gestures and characters in videos”, Multimedia Tools and Applications, An International Journal, ISSN 1380-7501, Volume 75, Number 19, pp.11615--11639. [Non-Patent Document 2] https: / / mediadrive.jp / products / et / [Non-Patent Document 3] "Spotting Method for Classification of Real World Data": Ryuichi Oka, The Computer Journal, Vol.41, No.8, pp.559-565 (1998). Summary of the Invention [Problem to be solved by the invention]

[0006] As a method for quantitatively expressing the above-mentioned actions of the caregiver, for example, an acceleration sensor that measures the movement of the caregiver in three-dimensional space may be used.

[0007] However, when performing measurements using acceleration sensors, the caregiver must wear multiple acceleration sensors at many points on his / her arms, legs, etc., which places a heavy burden on the caregiver. In addition, when multiple acceleration sensors are used, it is necessary to integrate the data measured by each acceleration sensor, making it difficult to easily identify the recognition of the caregiver's movements.

[0008] In contrast, a method for quantitatively evaluating the actions of a caregiver may involve, for example, capturing an image of the caregiver with an imaging device (camera) and recognizing the actions by using the captured video data. In this method, for example, a hidden Markov model (HMM), which is a method for recognizing time-series patterns, is used to perform category recognition based on vector features extracted from the video data.

[0009] However, since the HMM is a method mainly used in the field of voice recognition, it is not possible to accurately recognize the complex actions of a caregiver.

[0010] In addition, as a method for recognizing the movements of an object by using video data, for example, continuous DP (see Patent Document 1 and Non-Patent Document 1), which is a method for recognizing movement patterns generated in a single stroke, has already been proposed. However, in the case of an object that cannot be represented in a single stroke, such as a caregiver, where multiple points move simultaneously in a complex manner, the movements cannot be recognized with high accuracy.

[0011] Furthermore, there have been attempts to recognize the actions of caregivers using deep learning such as R-CNN (Recurrent Convolutional Neural Net). However, because R-CNN was developed with the purpose of targeting time series vectors, it is not able to recognize actions with high accuracy.

[0012] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide a motion identification program, an information processing device, and a motion identification method that enable a caregiver's motion to be quantitatively expressed. [Means for solving the problem]

[0013] To achieve the above-mentioned object, the motion identification program of the present invention causes a computer to execute a process of identifying first image data and second image data captured after the first image data from video data, calculating a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data, sorting each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, identifying motion characteristics of an object shown in the first image data and the second image data for each of the plurality of groups based on a length corresponding to each of the first optical flows for each of the plurality of pixels included in each group, and outputting information indicating the identified motion characteristics.

[0014] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that it identifies third image data captured after the second image data from the video data, calculates a second optical flow indicating a displacement for each of the plurality of pixels in the second image data and the third image data, assigns the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups based on an orientation corresponding to each of the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels, and identifies motion characteristics of the object for each of the plurality of groups based on a length corresponding to each of the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels included in each group.

[0015] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that it quantizes orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels, and assigns each of the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups such that optical flows having the same quantized orientation are assigned to the same group.

[0016] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that, for each of the multiple groups, a sum of lengths corresponding to the first optical flow for each of the multiple pixels included in each group and the second optical flow for each of the multiple pixels is calculated, and motion characteristics of the object are identified based on the sums corresponding to each of the multiple groups.

[0017] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that, for each of the multiple groups, a value is obtained by dividing the sum corresponding to each group by the elapsed time from the imaging time of the second image data to the imaging time of the third image data, and the value for each of the multiple groups is identified as the motion characteristic of the object.

[0018] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that it calculates, for each of the multiple groups, a first sum of lengths corresponding to the first optical flow for each of the multiple pixels included in each group and a second sum of lengths corresponding to the second optical flow for each of the multiple pixels included in each group, identifies, for each of the multiple groups, the number of sums among the first sums and the second sums corresponding to each group that is equal to or greater than a predetermined threshold, and identifies the number for each of the multiple groups as a motion characteristic of the object.

[0019] In addition, in one aspect, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that, for each of the multiple groups, a first sum of lengths corresponding to the first optical flow for each of the multiple pixels included in each group and a second sum of lengths corresponding to the second optical flow for each of the multiple pixels included in each group are calculated, and for each combination of a first group and a second group included in the multiple groups, an absolute value of a difference between the first sum corresponding to the first group and the second sum corresponding to the second group is calculated, and the absolute value for each combination of the first group and the second group is identified as a motion characteristic of the object.

[0020] In addition, the motion identification program of the present invention for achieving the above-mentioned object is characterized in that it causes a computer to execute a process of identifying first image data and second image data captured after the first image data from video data, calculating a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data, allocating each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, determining whether or not a first category for the motion of the object captured in the first image data and the second image data matches a second category for the motion of the object captured in the third image data and the fourth image data based on a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between third image data and fourth image data, and outputting information indicating whether or not the first category matches the second category.

[0021] In addition, in one aspect, the action identification program of the present invention for achieving the above-mentioned object is characterized in that it calculates a sum of the correlation values ​​for each of the multiple groups, determines whether the sum is equal to or greater than a predetermined threshold, and if it is determined that the sum is equal to or greater than the predetermined threshold, determines that the first category and the second category match.

[0022] In addition, in one aspect, the action identification program of the present invention for achieving the above-mentioned object is characterized in that it performs a predetermined blurring process on the first optical flows for each of the plurality of pixels, and assigns each of the first optical flows for each of the plurality of pixels to a plurality of groups based on the orientation corresponding to each of the first optical flows for each of the plurality of pixels after the predetermined blurring process.

[0023] In addition, in order to achieve the above-mentioned object, an information processing device of the present invention is characterized in having an image identification unit that identifies first image data and second image data captured after the first image data from video data, a flow calculation unit that calculates a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data, a flow allocation unit that distributes each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, a motion identification unit that identifies, for each of the plurality of groups, motion characteristics of an object captured in the first image data and the second image data based on a length corresponding to each of the first optical flows for each of the plurality of pixels included in each group, and an information output unit that outputs information indicating the identified motion characteristics.

[0024] In addition, in order to achieve the above-mentioned object, the information processing device of the present invention is characterized in having an image identification unit that identifies first image data and second image data captured after the first image data from video data; a flow calculation unit that calculates a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; a flow allocation unit that distributes each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels; a motion determination unit that determines whether or not a first category for the motion of an object captured in the first image data and the second image data and a second category for the motion of an object captured in the third image data and the fourth image data match based on a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between third image data and fourth image data; and an information output unit that outputs information indicating whether or not the first category and the second category match.

[0025] In addition, to achieve the above-mentioned object, the motion identification method of the present invention is characterized in that it causes a computer to execute processes of identifying first image data and second image data captured after the first image data from video data, calculating a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data, sorting each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, identifying motion characteristics of an object captured in the first image data and the second image data for each of the plurality of groups based on a length corresponding to each of the first optical flows for each of the plurality of pixels included in each group, and outputting information indicating the identified motion characteristics.

[0026] In addition, to achieve the above-mentioned object, the motion identification method of the present invention is characterized in that it has a computer execute a process of identifying first image data and second image data captured after the first image data from video data, calculating a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data, allocating each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, determining whether or not a first category for the motion of the object captured in the first image data and the second image data matches a second category for the motion of the object captured in the third image data and the fourth image data based on a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between third image data and fourth image data, and outputting information indicating whether or not the first category matches the second category. Effect of the Invention

[0027] According to the motion identification program, the information processing device, and the motion identification method of the present invention, it becomes possible to quantitatively express the motion of a caregiver. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an information processing device 1. As shown in FIG. [Diagram 2] FIG. 2 is a diagram illustrating the action identification process according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating details of the action identification process in the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating details of the action identification process in the first embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating details of the action identification process in the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating the optical flow and the directional pattern. [Figure 7] FIG. 7 is a diagram illustrating details of the action identification process in the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating details of the action identification process in the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating details of the action identification process in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating the action identification process according to the second embodiment. [Figure 11] FIG. 11 is a flowchart illustrating details of the action identification process in the second embodiment. [Figure 12] FIG. 12 is a flowchart illustrating details of the action identification process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the technical scope of the present invention is not limited to these embodiments.

[0030] [Configuration of information processing device] First, a description will be given of the hardware configuration of the information processing device 1. FIG 1 is a diagram for explaining the configuration of the information processing device 1.

[0031] 1, the information processing device 1 has a hardware configuration of a general-purpose computer device, and includes, for example, a CPU 101 which is a processor, a memory 102, a network interface 103, and a storage medium 104. Each unit is connected to each other via a bus 105.

[0032] The storage medium 104 has, for example, a program storage area (not shown) for storing a program (not shown) for performing a process (hereinafter also referred to as a motion identification process) for extracting the characteristics of a motion of a caregiver (hereinafter also referred to as a target object) shown in video data (image data). The storage medium 104 also has, for example, a storage area 110 for storing information used when performing the motion identification process. The storage medium 104 may be, for example, a HDD (Hard Disk Drive).

[0033] The CPU 101 executes a program loaded from the storage medium 104 (storage area 110) to the memory 102 to perform an operation specification process.

[0034] Furthermore, the network interface 103 communicates with, for example, the operation terminal 5.

[0035] [Movement Identification Processing in the First Embodiment] Next, a description will be given of the motion identification process in the first embodiment. Fig. 2 is a diagram for explaining the motion identification process in the first embodiment.

[0036] As shown in FIG. 2, the CPU 101 of the information processing device 1 functions as an image specifying unit 11, a flow calculating unit 12, a flow allocating unit 13, an action specifying unit 14, and an information output unit 15 by working in cooperation with a program.

[0037] The image specification unit 11 acquires, for example, video data stored in the storage area 110. The video data is, for example, video data captured in advance of a care work performed by a caregiver (care work that requires quantitative evaluation). Then, the image specification unit 11 specifies, for example, first image data included in the video data and second image data captured after the first image data.

[0038] The flow calculation unit 12 calculates, for example, a first optical flow indicating a displacement for each pixel in the first image data and the second image data specified by the image specification unit 11.

[0039] The flow allocating unit 13 allocates each of the first optical flows to a plurality of groups based on the direction in which the first optical flows calculated by the flow calculating unit 12 are facing, for example.

[0040] Specifically, the flow allocating unit 13, for example, quantizes the orientation corresponding to each of the first optical flows calculated by the flow calculating unit 12. Then, the flow allocating unit 13 allocates each of the first optical flows to a plurality of groups such that optical flows having the same quantized orientation are allocated to the same group.

[0041] The movement identification unit 14, for example, identifies (extracts) the movement characteristics of the caregiver shown in the first image data and the second image data based on the respective lengths of the first optical flows contained in each group, for each of the multiple groups assigned by the flow allocation unit 13.

[0042] The information output unit 15 outputs information indicating the motion characteristics identified by the motion identification unit 14 (motion characteristics of the caregiver).

[0043] That is, the information processing device 1 in the present embodiment performs grouping (quantization) of a plurality of optical flows so that optical flows with similar orientations are included in the same group. Then, the information processing device 1 performs grouping (quantization) of the caregiver's motion target for each optical flow included in each group.

[0044] This allows the information processing device 1 in the present embodiment to extract motion features that maintain information about the direction of the caregiver's motion. Therefore, the information processing device 1 can quantitatively express the motion of the caregiver by using the motion features extracted from each optical flow.

[0045] [Details of the Action Identification Process in the First Embodiment] Next, details of the action identification process in the first embodiment will be described below. Figures 3 to 9 are diagrams for explaining details of the action identification process in the first embodiment.

[0046] [Optical Flow] First, the optical flow will be described. Fig. 6 is a diagram for explaining the optical flow and the directional pattern.

[0047] Optical flow is a flow that indicates the displacement (movement distance and movement direction) of each pixel between image data contained in video data, and can be expressed as two-dimensional vector data, for example, as shown on the left side of Figure 6.

[0048] Specifically, the optical flow is used, for example, when recognizing the movement of an object contained in video data (image data).

[0049] [Character recognition using directional patterns] Next, character recognition (OCR: Optical Character Reader) using directional patterns (Non-Patent Document 2) will be described.

[0050] In character recognition using directional patterns, the Sobel method or the like is applied to image data (light and dark image data) of the character to be recognized to extract edges at each coordinate point contained in the image data (step 1). Here, the edges (two-dimensional vectors) extracted from the image data are expressed, for example, as in the following formula (1). In the following formula (1), x is the X coordinate of each coordinate point contained in the image data, y is the Y coordinate of each coordinate point contained in the image data, r is the magnitude of the vector, and θ is the orientation of the vector (angle from the X axis).

[0051]

number

[0052] Next, in character recognition using directional patterns, for example, a divided range that includes θ corresponding to the edge extracted from the image data is identified from among the eight ranges (hereinafter also referred to as divided ranges) shown in Figure 7, and θ is quantized to a value corresponding to the identified divided range (step 2).

[0053] Specifically, when θ corresponding to an edge extracted from image data is included in a division range corresponding to “θ=1”, θ corresponding to the edge is converted to 1.

[0054] In the following, a case where there are eight divided ranges will be described, but a number of divided ranges other than eight (for example, 16 divided ranges) may be provided.

[0055] Next, in character recognition using directional patterns, a two-dimensional scalar field is formed by arranging r corresponding to the edge corresponding to each divided range on the xy plane for each divided range (step 3). That is, for example, if there are eight divided ranges, eight two-dimensional scalar fields are formed.

[0056] Then, in character recognition using directional patterns, blurring is performed for each two-dimensional scalar field (step 4). In this case, for example, blurring is performed by convolution calculation using a Gaussian function, or blurring is performed by averaging multiple values ​​corresponding to the coordinates around each coordinate in the two-dimensional scalar field.

[0057] Here, while the edge of the character included in the image data can be expressed as in the above formula (1), the optical flow determined from the video data (plurality of image data) can be expressed, for example, as in the following formula (2): In the following formula (2), t indicates the time when each image data is captured.

[0058]

number

[0059] Therefore, it can be determined that character recognition using the above-mentioned directional patterns can also be applied to recognition of the actions of an object using optical flow. Therefore, in the action identification process in this embodiment, action recognition is performed using directional patterns, as described later.

[0060] [Flowchart of the Action Identification Process in the First Embodiment] 3 to 5 are flow charts for explaining details of the action identification process in the first embodiment. Note that the following will explain a case where optical flows corresponding to two sections (time periods) are calculated from three image data included in video data.

[0061] The image specification unit 11, for example, acquires the video data stored in the storage area 110. Then, as shown in Fig. 3, the image specification unit 11 specifies, from the video data, for example, first image data included in the video data, second image data captured after the first image data, and third image data captured after the second image data (S11).

[0062] Specifically, the image specification unit 11 acquires, for example, first image data captured at t=0, second image data captured at t=1, and third image data captured at t=2.

[0063] The first image data and the second image data may be consecutive image data or discontinuous image data in the moving image data stored in the storage area 110. Similarly, the second image data and the third image data may be consecutive image data or discontinuous image data in the moving image data stored in the storage area 110.

[0064] Alternatively, the image specification unit 11 may sequentially specify image data from the video data, and continuously perform the processes from S12 onward on each of the specified image data.

[0065] Then, the flow calculation unit 12 calculates a first optical flow indicating a displacement for each pixel in the first image data and the second image data identified in the process of S11 (S12).

[0066] Specifically, as shown on the right side of FIG. 6, the flow calculation unit 12 calculates a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data.

[0067] Furthermore, the flow calculation unit 12 calculates a second optical flow indicating a displacement for each pixel in the second image data and the third image data identified in the process of S11 (S13).

[0068] That is, in this case, the flow calculation unit 12 calculates a first optical flow corresponding to the section between the first image data and the second image data, and a second optical flow corresponding to the section between the second image data and the third image data.

[0069] Specifically, the flow calculation unit 12 expresses each optical flow by using, for example, the above formula (2).

[0070] In the following, in the processes of S12 and S13, a case where optical flows corresponding to two sections are calculated will be described, but optical flows corresponding to three or more sections may be calculated.

[0071] Next, the flow allocating unit 13 quantizes the directions corresponding to the first optical flow and the second optical flow calculated in the processes of S12 and S13 (S14).

[0072] Specifically, the flow allocator 13 identifies a divided range that includes θ corresponding to each optical flow among, for example, the eight divided ranges shown in FIG. 7, and converts θ to a value corresponding to the identified divided range.

[0073] Then, the flow allocating unit 13 allocates each of the first optical flow and the second optical flow to a plurality of groups so that optical flows having the same quantized value in the process of S14 are allocated to the same group (S15).

[0074] Specifically, the flow allocating unit 13 identifies, for each divided range, optical flows whose quantized values ​​are included in the divided range. Then, the flow allocating unit 13 groups the optical flows so that optical flows corresponding to the same divided range are included in the same group.

[0075] Next, the motion identification unit 14 calculates, for each of the groups, the sum of lengths corresponding to the first optical flows included in each group (hereinafter, also referred to as the first sum) (S21), as shown in Fig. 4. Specifically, the motion identification unit 14 calculates, for each of the groups, the sum of values ​​corresponding to the directional patterns of the first optical flows included in each of the groups (the first sum).

[0076] Furthermore, the motion identification unit 14 calculates, for each of the groups, the sum of lengths corresponding to the second optical flows included in each group (hereinafter also referred to as the second sum) (S22). Specifically, the motion identification unit 14 calculates, for each of the groups, the sum of values ​​corresponding to the directional patterns of the second optical flows included in each of the groups (the second sum). The processes of S21 and S22 will be described in detail below.

[0077] [Details of S21 and S22 processing] The motion identification unit 14 first calculates a directional pattern corresponding to each of the first optical flow and the second optical flow by using the following formula (3). In the following formula (3), F(x, y, θ':t) indicates the directional pattern, θ' indicates each of the divided ranges, and T indicates the time difference between the imaging times of each image data used to calculate the optical flow. Note that δ(θ'-θ) is a delta function that takes the value 1 when θ' and θ match, and takes the value 0 when θ' and θ do not match.

[0078]

number

[0079] That is, in formula (3), the value corresponding to the divided range that does not include the orientation of each optical flow is 0. On the other hand, in formula (3), the value corresponding to the divided range that includes the orientation of each optical flow is a value other than 0.

[0080] Furthermore, the motion identification unit 14 calculates a first sum and a second sum corresponding to each of the groups by using the following formula (4): In the following formula (4), K(θ′,t) represents the sum (first sum and second sum) of values ​​corresponding to the directional patterns of the multiple optical flows included in each group.

[0081]

number

[0082] When optical flows corresponding to three or more sections are calculated in the processes of S12 and S13, the motion identification unit 14 calculates sums corresponding to the three or more sections in the processes of S21 and S22.

[0083] Next, the motion identification unit 14 calculates, for example, for each of a plurality of groups, a total value of the sums corresponding to a plurality of optical flows calculated from image data captured from a start time to an end time. The start time indicates the start time of the time period during which the motion characteristics of the caregiver are identified, and the end time indicates the start time of the time period during which the motion characteristics of the caregiver are identified. The time from the start time to the end time is, for example, about several seconds to several minutes. In this case, the motion identification unit 14 may divide the calculated total value by the time from the start time to the end time.

[0084] Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 calculates the sum of the first sum and the second sum corresponding to each group for each group (S23). Then, for example, the motion identification unit 14 divides the calculated sum by the elapsed time from the imaging time of the second image data to the imaging time of the third image data for each group.

[0085] More specifically, the motion identification unit 14 calculates the value by using, for example, the following formula (5). In the following formula (5), t start indicates the start time, and t end indicates the end time.

[0086]

number

[0087] Then, the motion identification unit 14 identifies each of the total values ​​for the multiple groups calculated in the process of S23 as a first motion feature of the target object (S24). That is, the first motion feature is, for example, an index indicating an average amount of motion by a caregiver.

[0088] Next, the motion identification unit 14 identifies, for each of the groups, the number of sums that are equal to or greater than a predetermined threshold value among the sums corresponding to the optical flows calculated from the image data captured from the start time to the end time. In this case, the motion identification unit 14 may divide the number of the identified sums by the time from the start time to the end time.

[0089] Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 identifies, for each of the multiple groups, the number of sums equal to or greater than a predetermined threshold among the first sums and the second sums corresponding to each of the multiple groups (S31), as shown in Fig. 5. Then, for example, the motion identification unit 14 divides, for each of the multiple groups, the number of the identified sums by the elapsed time from the imaging time of the second image data to the imaging time of the third image data.

[0090] More specifically, the motion identification unit 14 calculates the number of sums equal to or greater than a predetermined threshold by using, for example, the following formula (6): In the following formula (6), h represents a threshold, and 1[X] represents a function that takes the value 1 when X is equal to or greater than 0, and takes the value 0 when X is less than 0.

[0091]

number

[0092] Then, the motion identification unit 14 identifies each of the total numbers for each of the multiple groups identified in the process of S31 as a second motion feature of the target object (S32). That is, the second motion feature is, for example, an index indicating the number of times the caregiver performed a motion with a motion amount larger than a threshold.

[0093] Next, the motion identification unit 14 calculates, for example, for each combination of two groups (hereinafter also referred to as the first group and the second group) included in the multiple groups, a sum corresponding to the first group among the sums corresponding to the multiple optical flows calculated from image data captured from the start time to the end time (hereinafter also referred to as the specific image data). Also, for example, for each combination of the first group and the second group, the motion identification unit 14 calculates, for example, a sum corresponding to the second group among the sums corresponding to the multiple optical flows calculated from other image data captured a predetermined time after the specific image data. Then, for example, for each combination of the first group and the second group, the motion identification unit 14 calculates a sum value of the absolute value of the difference between the sums corresponding to the first group and the sums corresponding to the second group. In this case, the motion identification unit 14 may divide the calculated sum value by the time from the start time to the end time.

[0094] Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 calculates the absolute value of the difference between the first sum corresponding to the first group and the second sum corresponding to the second group (the sum of the absolute values) for each combination of the first group and the second group included in the multiple groups (S33). Then, for example, the motion identification unit 14 divides the calculated absolute value (the sum of the absolute values) for each combination of the first group and the second group by the elapsed time from the imaging time of the second image data to the imaging time of the third image data.

[0095] More specifically, the motion identification unit 14 calculates the sum of the absolute values ​​of the differences between the sums corresponding to the first group and the sums corresponding to the second group by using, for example, the following formula (7): In the following formula (7), θ' indicates the division range corresponding to the first group, Δθ' indicates the division range corresponding to the second group, and τ indicates a predetermined time.

[0096]

number

[0097] Then, the motion identification unit 14 identifies each absolute value for each combination of two groups included in the groups calculated in the process of S33 as a third motion feature of the target object (S34). That is, the third motion feature is, for example, an index indicating the amount of angular displacement of the motion by the caregiver.

[0098] After that, the information output unit 15 outputs, for example, at least one of the first action characteristic, the second action characteristic, and the third action characteristic to the operation terminal 5 (S35).

[0099] Specifically, the information output unit 15 outputs the first action features corresponding to each group (a group corresponding to each of the eight divided ranges) in a state in which the first action features are distributed in a regular octagonal graph, for example, as shown in FIGS. 8 and 9.

[0100] This enables the information processing device 1 to output each index (the first motion feature, the second motion feature, and the third motion feature) that quantitatively expresses the motion features of the caregiver.

[0101] In this regard, the first, second and third motion characteristics are indices calculated by integrating values ​​distributed in the scalar field for each group, and therefore do not have coordinate information in the scalar field. Therefore, the information processing device 1 can output the motion patterns of the caregiver in a state where they are easily visualized.

[0102] In the above example, the case where the motion characteristics of a caregiver are extracted has been described, but the information processing device 1 may extract motion characteristics in the field of sports, for example.

[0103] [Action Identification Processing in the Second Embodiment] Next, the motion identification process in the second embodiment will be described with reference to Fig. 10 which is a diagram for explaining the motion identification process in the second embodiment.

[0104] As shown in FIG. 10, the CPU 101 of the information processing device 1 functions as an image specifying unit 21, a flow calculating unit 22, a blurring processing unit 23, a flow allocating unit 24, an action determining unit 25, and an information output unit 26 by working in cooperation with a program.

[0105] The image specification unit 21, for example, acquires moving image data stored in the storage area 110. Then, the image specification unit 21 specifies, for example, first image data included in the moving image data and second image data captured after the first image data.

[0106] The flow calculation unit 22 calculates, for example, a first optical flow indicating a displacement for each pixel between the first image data and the second image data specified by the image specification unit 21.

[0107] The blurring processing unit 23 performs a predetermined blurring process on the first optical flow calculated by the flow calculation unit 22, for example.

[0108] Specifically, the blurring processor 23 performs blurring on the first optical flow using a Gaussian function, for example.

[0109] The flow allocating unit 24 allocates each of the first optical flows, which have been blurred by the blurring unit 23, into a plurality of groups, based on the direction in which the first optical flows are facing.

[0110] Specifically, the flow allocating unit 24, for example, quantizes the orientation corresponding to each of the first optical flows that have been blurred by the blurring processing unit 23. Then, the flow allocating unit 24 allocates each of the first optical flows to a plurality of groups such that optical flows having the same quantized orientation are allocated to the same group.

[0111] The movement determination unit 25 calculates, for example, for each of the groups sorted by the flow sorting unit 24, a correlation value between a length corresponding to each of the first optical flows included in each group and a length corresponding to each of the second optical flows indicating a displacement for each pixel between the third image data and the fourth image data prepared in advance. Specifically, the movement determination unit 25 calculates, for example, for each of the groups, a correlation value between a directional pattern corresponding to each of the first optical flows included in each group and a directional pattern corresponding to each of the second optical flows prepared in advance. Then, the movement determination unit 25 determines, for example, from the correlation value for each of the groups sorted by the flow sorting unit 24, whether or not a category (hereinafter also referred to as a first category) of the movement of the object shown in the first image data and the second image data matches a category (hereinafter also referred to as a second category) of the movement of the object shown in the third image data and the fourth image data.

[0112] The information output section 26 outputs, for example, the result of the determination made by the action determination section 25 (information indicating whether or not the first category and the second category match).

[0113] That is, the information processing device 1 in this embodiment calculates a correlation value between a directionality pattern of a newly calculated first optical flow and a directionality pattern of a second optical flow whose category of motion is known in advance, for example. If the calculated correlation value satisfies a predetermined condition, the information processing device 1 determines that the first category corresponding to the first optical flow matches the second category (the category known in advance) corresponding to the second optical flow.

[0114] This enables the information processing device 1 to recognize, for example, a category of the action of the caregiver shown in each image data.

[0115] [Details of the Action Identification Process in the Second Embodiment] Next, details of the action identification process in the second embodiment will be described below. Figures 11 and 12 are flow charts illustrating details of the action identification process in the second embodiment.

[0116] The image specification unit 21, for example, acquires the video data stored in the storage area 110. Then, as shown in Fig. 11, the image specification unit 11 specifies, for example, first image data included in the video data and second image data captured after the first image data from the video data (S41).

[0117] Then, the flow calculation section 22 calculates a first optical flow indicating the displacement for each pixel between the first image data and the second image data identified in the process of S41 (S42).

[0118] Next, the flow allocating unit 24 quantizes the directions corresponding to the first optical flows calculated in the process of S42 (S43).

[0119] Thereafter, the blurring processor 23 performs a predetermined blurring process on the first optical flow quantized in the process of S43 (S44). A specific example of the process of S44 will be described below.

[0120] [Specific example of S44 processing] The blurring processor 23 performs a process of blurring the coordinates (x, y) in the two-dimensional scalar field of the directional pattern expressed by the above formula (3) to a nearby value. This enables the blurring processor 23 to spatially expand the value of θ in formula (3), and allows the feature of θ to exist even at nearby points. This is different from the blurring process that spatially averages vectors between each optical flow and other optical flows located at coordinates around each optical flow. Therefore, the information processing device 1 can improve the recognition accuracy of categories regarding the actions of the caregiver.

[0121] Specifically, for example, when the Gaussian function is expressed as in the following formula (8), the directional pattern obtained by performing blurring processing using the Gaussian function is expressed as in the following formula (9). In the following formula (8), σ represents a parameter. In addition, in the following formula (9), G(x, y, θ:t) represents the directional pattern obtained by performing blurring processing using the Gaussian function, i represents the X coordinate of each coordinate point included in the image data, and j represents the Y coordinate of each coordinate point included in the image data.

[0122]

number

[0123]

number

[0124] In addition, when the pixel size of (x, y) is different from the pixel size of (i, j), for example, when the pixel size of (x, y) is 512 × 512 and the pixel size of (i, j) is 32 × 32, the directional pattern obtained by blurring using a Gaussian function can be expressed as shown in the following equation (10).

[0125]

number

[0126] Next, the flow allocating unit 24 allocates the first optical flows that have been blurred in the process of S44 to a plurality of groups so that optical flows having the same orientation quantized in the process of S43 are allocated to the same group (S51).

[0127] Then, the motion judgment unit 25 calculates, for example, for each of the multiple groups sorted in the processing of S51, a correlation value between the length of the first optical flow that has been blurred in the processing of S44 and the length of the second optical flow that indicates the displacement for each pixel between the third image data and the fourth image data, which are reference image data prepared in advance (S52).

[0128] Specifically, the operation determination unit 25 calculates a correlation value between the directional pattern of the first optical flow and the directional pattern of the second optical flow for each of the groups divided in the process of S51.

[0129] Next, the operation determining section 25 calculates the sum of the correlation values ​​for each of the multiple groups sorted in the process of S51 (S53).

[0130] Furthermore, the action determination section 25 determines whether or not the first category and the second category match, depending on whether or not the sum calculated in the process of S53 is equal to or greater than a predetermined threshold (S54).

[0131] Thereafter, the information output unit 26 outputs information indicating the determination result as to whether or not the first category and the second category match (S55). A specific example of the processes from S52 to S54 will be described below.

[0132] [Specific example of processing from S52 to S54 (1)] First, a first specific example of the processes from S52 to S54 will be described.

[0133] The first specific example is an example in which a category corresponding to a directional pattern of a first optical flow whose category of motion is unknown is identified by using a directional pattern of a second optical flow whose category of motion is known. Hereinafter, the directional pattern of the second optical flow whose category of motion is known is also called a reference directional pattern, and the directional pattern of the first optical flow whose category of motion is unknown is also called an unknown directional pattern.

[0134] In the process of S52, the operation determination unit 25 calculates a cosine correlation value between the reference directivity pattern and the unknown directivity pattern for each of the multiple groups. Specifically, the operation determination unit 25 calculates the cosine correlation value for each of the multiple groups by using, for example, the following equation (11). In the following equation (11), ρ(G(θ:t),Z(θ:t fixed )) denotes the cosine correlation value, G(i,j,θ:t) denotes the unknown directional pattern, and Z(i,j,θ:t fixed ) indicates the reference directional pattern.

[0135]

number

[0136] Then, in the process of S53, the operation determination unit 25 calculates the sum of the cosine correlation values ​​for each of the groups by using, for example, the following equation (12). In the following equation (12), ρ(G(t),Z(t fixed )) indicates the sum of the cosine correlation values.

[0137]

number

[0138] Furthermore, in the processing of S54, for example, if the sum of the cosine correlation values ​​calculated in the processing of S53 is equal to or greater than a predetermined threshold value, the movement determination unit 25 determines that the category for the movement corresponding to the reference directivity pattern and the category for the movement corresponding to the unknown directivity pattern match.

[0139] [Specific example of processing from S52 to S54 (2)] Next, a second specific example of the process from S52 to S54 will be described.

[0140] The first specific example is an example of identifying a category corresponding to an unknown directional pattern of a first optical flow calculated from two image data (first image data and second image data) by using a reference directional pattern of a second optical flow calculated from two image data (third image data and fourth image data). In contrast, the second specific example is an example of identifying a category of a movement corresponding to a series of unknown directional patterns by determining whether or not a reference directional pattern of a second optical flow corresponding to a plurality of sections (time periods) calculated from three or more image data (hereinafter also referred to as a series of reference directional patterns) is included in an unknown directional pattern of a first optical flow corresponding to a plurality of sections calculated from three or more image data (hereinafter also referred to as a series of unknown directional patterns).

[0141] In the process of S52, the operation determining unit 25 calculates the cosine correlation value between the sequence of the reference directivity pattern and the sequence of the unknown directivity pattern by using, for example, the above equation (11).

[0142] Then, in the process of S53, the action determining section 25 calculates the sum of the cosine correlation values ​​for each of the multiple groups, for example, by using the above formula (12).

[0143] Then, in the processing of S54, the action determination unit 25 uses continuous DP (see Patent Document 1, Non-Patent Document 1, and Non-Patent Document 3) to determine whether or not a category for an action corresponding to a sequence of reference directional patterns is included in a category corresponding to a sequence of unknown directional patterns.

[0144] Specifically, the operation determination unit 25 calculates Q(t) by using, for example, the following equation (13). In the following equation (13), τ is t in the above equation (12). fixed It supports.

[0145]

number

[0146] Then, when there is a point in time when Q(t) is greater than or equal to a predetermined threshold and becomes a local maximum, the movement determination unit 25 determines that the sequence of reference directional patterns is included in the sequence of unknown directional patterns, and recognizes the category of the movement corresponding to the sequence of reference directional patterns as the category of the movement corresponding to the sequence of unknown directional patterns.

[0147] That is, in the second specific example, by using continuous DP, matching is performed using local correlation values ​​between a sequence of reference directional patterns and a sequence of unknown directional patterns, and a category for the action corresponding to the sequence of unknown directional patterns is identified. [Explanation of symbols]

[0148] 1: Information processing device 5: Operation terminal 11: Image identification section 12: Flow calculation section 13: Flow distribution unit 14: Operation specific part 15: Information output section 21: Image identification section 22: Flow calculation unit 23: Blur processing section 24: Flow distribution unit 25: Operation specific part 26: Information output section 101:CPU 102: Memory 103: Network interface 104:Storage medium 105: Bus

Claims

1. Identifying first image data and second image data captured after the first image data from the video data; Identifying third image data captured after the second image data from the video data; Calculating a first optical flow indicative of a displacement for each of a plurality of pixels in the first image data and the second image data; calculating a second optical flow indicative of a displacement for each of the plurality of pixels in the second image data and the third image data; sorting the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels into a plurality of groups based on orientations corresponding to the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels; calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group; identifying, for each of the plurality of groups, the number of sums that are equal to or greater than a predetermined threshold among the first sums and the second sums corresponding to each group; Identifying the number of each of the plurality of groups as a motion characteristic of the object captured in the first image data and the second image data; outputting information indicating the identified operational characteristics; An action specifying program that causes a computer to execute a process.

2. In claim 1, In the allocation process, quantizing orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels; assigning the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups such that optical flows having the same quantized orientation are assigned to the same group; 2. An action specifying program comprising:

3. In claim 2, In the identifying process, calculating a sum of the first sum and the second sum corresponding to each of the plurality of groups; identifying a motion characteristic of the object based on the sum values ​​corresponding to each of the plurality of groups; 2. An action specifying program comprising:

4. In claim 3, In the identifying process, For each of the plurality of groups, the sum corresponding to each group is calculated based on the second image data. a value calculated by dividing the image time by the elapsed time from the image time of the third image data; identifying the values ​​for each of the plurality of groups as motion characteristics of the object; 2. An action specifying program comprising:

5. In claim 2, In the identifying process, Calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flow for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flow for each of the plurality of pixels included in each group; calculating an absolute value of a difference between the first sum corresponding to the first group and the second sum corresponding to the second group for each combination of a first group and a second group included in the plurality of groups; identifying the absolute value for each combination of the first group and the second group as a motion characteristic of the object; 2. An action specifying program comprising:

6. an image specification unit that specifies, from video data, first image data and second image data captured after the first image data, and specifies, from the video data, third image data captured after the second image data; a flow calculation unit that calculates a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data, and calculates a second optical flow indicating a displacement for each of the plurality of pixels in the second image data and the third image data; a flow allocation unit that allocates the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to a plurality of groups based on orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels; a motion identification unit that calculates, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group, identifies, for each of the plurality of groups, the number of the first sums and the second sums corresponding to each of the plurality of groups that is equal to or greater than a predetermined threshold, and identifies the number for each of the plurality of groups as a motion characteristic of an object captured in the first image data and the second image data; and an information output unit that outputs information indicating the identified operation characteristic.

23. An information processing apparatus comprising:

7. Identifying first image data and second image data captured after the first image data from the video data; Identifying third image data captured after the second image data from the video data; Calculating a first optical flow indicative of a displacement for each of a plurality of pixels in the first image data and the second image data; calculating a second optical flow indicative of a displacement for each of the plurality of pixels in the second image data and the third image data; sorting the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels into a plurality of groups based on orientations corresponding to the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels; calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group; identifying, for each of the plurality of groups, the number of sums that are equal to or greater than a predetermined threshold among the first sums and the second sums corresponding to each group; Identifying the number of each of the plurality of groups as a motion characteristic of the object captured in the first image data and the second image data; outputting information indicating the identified operational characteristics; A motion identification method comprising the steps of: causing a computer to execute a process.

Images