Image processing system, image processing method, and program

Abstract

An image processing system (10) comprises an analyzing means (11), a determining means (12), and a label assigning means (13). The analyzing means (11) recognizes, from image data of motion images over a plurality of consecutive frames in which a person performing a series of motion is captured, a plurality of motion images indicating the person's motion. The determining means (12) determines whether the motion image and a predetermined reference motion are associated with each other. The label assigning means (13) assigns, on the basis of the determination, a label to at least some consecutive frames within the motion image.

Description

Technical Field

[0001] The present disclosure relates to an image processing system, an image processing method, and a computer-readable medium.

Background Art

[0002] Techniques for classifying the actions of a person included in image data have been developed.

[0003] For example, the technique of Patent Document 1 detects the state information of the face and hands of a target person in a moving image of the target person, and determines the type of action of the target person based on the similarity between the state information and an action model.

[0004] The person action detection device according to the technique of Patent Document 2 generates the trajectory of feature points as feature point trajectory information for each frame image of a video, and accumulates the direction and magnitude of the movement vector of the feature points for each range width obtained by dividing the possible range thereof into a predetermined number to generate a trajectory feature quantity. Further, the person action detection device generates a distribution in which clusters to which the trajectory feature quantities belong are accumulated from a plurality of trajectory feature quantities within a predetermined time interval, and compares the distribution with learning data to identify the action of a person.

[0005] The technique of Patent Document 3 extracts skeleton information based on the joints of a person from video data in time series, extracts the surrounding area of the skeleton information, recognizes an action from the surrounding area of the video data, and outputs an integrated score for each action.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

[0007] However, while the aforementioned technology can recognize individual actions, it cannot process images containing multiple actions to separate them into individual actions.

[0008] In view of the above-mentioned issues, the purpose of this disclosure is to provide an image processing system, etc., that can efficiently separate and process human movements from images captured by a camera. [Means for solving the problem]

[0009] An image processing system according to one aspect of this disclosure includes an analysis means, a determination means, and a labeling means. The analysis means recognizes a plurality of motion images representing a person's actions from image data relating to a plurality of consecutive frames of a person performing a series of actions. The determination means determines whether or not the motion images are related to a predetermined reference action. The labeling means assigns labels to at least a portion of the consecutive frames of the motion images based on the determination.

[0010] An image processing method according to one aspect of the present disclosure involves a computer performing the following processes: The computer recognizes multiple action images representing a person's actions from image data relating to multiple consecutive frames of a person performing a series of actions. The computer determines whether the action images are associated with a predetermined reference action. Based on the determination, the computer assigns labels to at least some of the consecutive frames of the action images.

[0011] A computer-readable medium according to one aspect of the present disclosure causes a computer to perform the following image processing method: The computer recognizes multiple action images representing a person's actions from image data relating to multiple consecutive frames of a person performing a series of actions. The computer determines whether the action images are associated with a predetermined reference action. Based on the determination, the computer assigns labels to at least some of the consecutive frames of the action images.

Advantages of the Invention

[0012] According to the present disclosure, an image processing system or the like that can efficiently perform the segmentation process of a person's motion from an image captured by a camera can be provided.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram showing the configuration of the image processing system according to Embodiment 1. [Figure 2] It is a flowchart showing the image processing method according to Embodiment 1. [Figure 3] It is a diagram showing the overall configuration of the image processing system according to Embodiment 2. [Figure 4] It is a diagram showing the skeleton data extracted from the image data. [Figure 5] It is a diagram for explaining the reference motion data according to Embodiment 2. [Figure 6] It is a diagram for explaining the first example of the reference motion according to Embodiment 2. [Figure 7] It is a diagram for explaining the second example of the reference motion according to Embodiment 2. [Figure 8] It is a diagram for explaining an example of the label data. [Figure 9] It is a block diagram showing the overall configuration of the image processing system according to Embodiment 3. [Figure 10] It is a diagram for explaining the skeleton data of the upper limb. [Figure 11] It is a flowchart showing the image processing method according to Embodiment 3. [Figure 12] It is a diagram for explaining the reference motion data according to Embodiment 3. [Figure 13] It is a diagram showing the first example of the image including the label data according to Embodiment 3. [Figure 14] It is a diagram showing the second example of the image including the label data according to Embodiment 3. [Figure 15]It is a diagram showing a third example of an image including label data according to Embodiment 3. [Figure 16] It is a block diagram illustrating the hardware configuration of a computer.

Mode for Carrying Out the Invention

[0014] Hereinafter, the present disclosure will be described through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. In each drawing, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted as necessary.

[0015] <Embodiment 1> First, Embodiment 1 of the present disclosure will be described. FIG. 1 is a block diagram showing the configuration of an image processing system 10 according to Embodiment 1. The image processing system 10 shown in FIG. 1 analyzes, for example, the posture and movement of a person included in an image captured by a camera, and assigns a label that can classify the movement of the person in the image. The main components of the image processing system 10 are an analysis unit 11, a determination unit 12, and a labeling unit 13. In the present disclosure, "posture" refers to the form of at least a part of the body, and "movement" refers to the state of taking a predetermined posture over time. "Movement" is not limited to the case where the posture changes, and also includes the case where a certain posture is maintained. Therefore, when simply referring to "movement", it may include the posture.

[0016] The analysis unit 11 recognizes multiple motion images showing a person's movements from predetermined image data. The predetermined image data is image data covering multiple consecutive frames of a person performing a series of movements. The image data is, for example, image data in a predetermined format such as H.264 or H.265. The series of movements can be any movements, but it is preferable that it includes a predetermined posture or movement that can be classified. For example, the series of movements could be a predetermined task, dance, exercise, and manners. A "motion image" is an image that contains information that allows the person's movements to be classified. That is, a motion image may be the image of the person's body itself taken by a predetermined camera, or it may be the image data of the person taken by the camera that has been cropped, brightness adjusted, or enlarged or reduced. A motion image may also be an image showing the person's movements estimated by analyzing the image of the person taken by the camera.

[0017] The analysis unit 11 recognizes skeletal data relating to the structure of a person's body included in predetermined image data as a motion image. Here, skeletal data is data indicating the structure of a person's body for detecting the person's posture or movement, and is composed of a combination of multiple pseudo joint points and pseudo skeletal structures.

[0018] The determination unit 12 determines whether the motion image recognized by the analysis unit 11 is related to a predetermined reference motion. Alternatively, the determination unit 12 can determine whether the motion shown in each time-series interval of the image data is similar to the predetermined reference motion. In this case, the determination unit 12 uses reference skeleton data relating to the reference motion for comparison with the skeleton data, which is the motion image. The reference skeleton data is pre-set skeleton data, which may be possessed by the image processing system 10 in advance, or which the image processing system 10 may acquire from an external source.

[0019] Reference skeletal data is, for example, skeletal data extracted from predetermined reference image data. The reference image data used to extract skeletal data may be image data containing a single frame image, or it may be image data containing multiple consecutive frames taken at multiple different times in a video. In the following explanation, an image for one frame may be referred to as a frame image or simply a frame.

[0020] When determining whether an action image and a reference action are related, the determination unit 12 calculates the similarity between the skeletal data relating to the action image and predetermined reference skeletal data. For example, if this similarity is greater than or equal to a predetermined value, the determination unit 12 determines that the recognized action image and the reference action are related. Conversely, if this similarity is less than a predetermined value, the determination unit 12 does not determine that the recognized action image and the reference action are related.

[0021] The labeling unit 13 assigns labels to at least a portion of consecutive frames within the motion image, according to the determination made by the determination unit 12. In other words, the labeling unit 13 generates label data corresponding to the motion image of a person performing a series of actions. The label data is data configured to allow for the identification of a predetermined reference action from the motion image. The label data is configured to correspond to the time series of the motion image. This enables the image processing system 10 to perform the identification of a predetermined reference action from the actions of a person included in the motion image. In other words, the labeling unit 13 assigns a label to a section in the image data indicating the type of action represented by that section, based on the determination made by the determination unit 12.

[0022] Next, the processing of the image processing system 10 will be described with reference to Figure 2. Figure 2 is a flowchart showing the flow of the image processing method according to Embodiment 1. The flowchart shown in Figure 2 starts, for example, when the image processing system 10 acquires image data.

[0023] First, the analysis unit 11 recognizes multiple motion images representing a person's actions from image data of multiple consecutive frames capturing a person performing a series of actions (step S11). Once the analysis unit 11 recognizes motion images, it supplies information about the recognized motion images to the determination unit 12.

[0024] Next, the determination unit 12 determines whether the motion image and a predetermined reference motion are related (step S12). More specifically, the determination unit 12 compares the motion of the person in the motion image with the reference motion and determines whether they are similar. The determination unit 12 supplies information regarding the result of this determination to the labeling unit 13.

[0025] Next, the labeling unit 13 assigns labels to at least a portion of consecutive frames in the motion image based on the determination made by the determination unit 12 (step S13). That is, the labeling unit 13 assigns labels to frames in the image data that contain motions that it has determined to be similar to the reference motion. Once the labeling unit 13 has assigned labels to the target frames in the image data, the image processing system 10 terminates the series of processes.

[0026] Embodiment 1 has been described above. The image processing system 10 also includes a processor and a storage device, although these are not shown in the diagram. The storage device of the image processing system 10 includes, for example, a non-volatile memory such as flash memory or an SSD (Solid State Drive). In this case, the storage device of the image processing system 10 stores a computer program (hereinafter also simply referred to as "the program") for executing the image processing method described above. The processor loads the computer program from the storage device into a buffer memory such as DRAM (Dynamic Random Access Memory) and executes the program.

[0027] Each component of the image processing system 10 may be implemented with dedicated hardware. Furthermore, some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be implemented by a single chip or by multiple chips connected via a bus. Some or all of each component of each device may be implemented by a combination of the aforementioned circuits, etc., and programs. Furthermore, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (field-programmable gate array), etc., can be used as the processor. Note that the descriptions of the configurations described herein may also apply to other devices or systems described below in this disclosure.

[0028] Furthermore, if some or all of the components of the image processing system 10 are implemented by multiple information processing devices or circuits, these devices may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. The image processing method executed by the image processing system 10 may also be provided in SaaS (Software as a Service) format. The image processing method described above may also be stored on a computer-readable medium in order to have a computer execute the method.

[0029] According to this embodiment, an image processing system that can efficiently separate and process human movements from images captured by a camera can be provided.

[0030] <Embodiment 2> Next, Embodiment 2 will be described. Figure 3 is a diagram showing the overall configuration of the image processing system according to Embodiment 2. Figure 3 shows a camera 100 and an information processing system 200. The camera 100 is installed to capture images of a person P10. The camera 100 is connected to a network N1 and supplies image data of the captured images to the information processing system 200 via the network N1.

[0031] The information processing system 200 analyzes images related to the movements of person P10 contained in the image data received from camera 100. The information processing system 200 mainly consists of an image data acquisition unit 201, a motion analysis unit 202, an image data storage unit 203, and an image processing system 20.

[0032] The image data acquisition unit 201 acquires image data supplied from the camera 100 and supplies the acquired image data to the image data storage unit 203. The motion analysis unit 202 analyzes the image data to which labels have been assigned by the image processing system 20. The processing performed by the motion analysis unit 202 may be arbitrary. For example, the processing performed by the motion analysis unit 202 may include statistical processing such as counting the number of frames for images containing a predetermined type of motion. The information processing system 200 may also have a display unit (not shown) that displays the images related to the motion analysis and presents them to the user. The motion analysis unit 202 may also assist the user in analyzing the image data by displaying the image data and data related to the labels assigned by the image processing system 20.

[0033] The image data storage unit 203 is a storage device that includes non-volatile memory such as flash memory, SSD, or HDD. The image data storage unit 203 receives image data acquired by the image data acquisition unit 201 and stores the received image data. The image data storage unit 203 also stores data related to labels (label data) generated by the image processing system 20.

[0034] The image processing system 20 assigns labels to image data in order to separate motion images related to human movement in the image data acquired by the information processing system 200. The image processing system 20 differs from the image processing system 10 according to Embodiment 1 in that it has a reference motion data storage unit 14.

[0035] In this embodiment, the analysis unit 11 extracts images of a person's body from the image data and generates skeletal data relating to the structure of the person's body from the extracted image data. This allows the analysis unit 11 to recognize motion images related to the person's movements from the image data.

[0036] In this embodiment, the determination unit 12 uses the shapes of the elements constituting the skeletal data to determine whether the skeletal data relating to a person's movement and the skeletal data representing a reference movement are similar. The determination unit 12 also determines that the movement image relating to a person's movement is related to the reference movement if the movement image relating to the person's movement and the skeletal data representing the reference movement are similar.

[0037] In this embodiment, the labeling unit 13 assigns a label to frames of motion images that are associated with a predetermined reference operation, indicating that they are related to the reference operation. More specifically, the labeling unit 13 generates label data and supplies the generated label data to the image data storage unit 203.

[0038] The reference operation data storage unit 14 is one form of storage means of the image processing system 20, and stores a plurality of predetermined reference operations in an updatable manner. In this case, when the reference operation data is updated, the determination unit 12 uses the updated data relating to the reference operation to perform the above determination.

[0039] Next, an example of detecting human movement will be described with reference to Figure 4. Figure 4 shows skeletal data of a body extracted from image data. The image shown in Figure 4 is a body image F10 extracted from an image taken by camera 100 of person P10. In the image processing system 10, the analysis unit 11 cuts out the body image F10 from the image taken by camera 100 and further sets the skeletal structure.

[0040] The analysis unit 11 extracts, for example, feature points from the image that could be key points of person P10. Furthermore, the analysis unit 11 detects key points from the extracted feature points. When detecting key points, the analysis unit 11 refers to, for example, machine learning information about the key point image.

[0041] In the example shown in Figure 4, the analysis unit 11 detects the head A1, neck A2, right shoulder A31, left shoulder A32, right elbow A41, left elbow A42, right hand A51, left hand A52, right hip A61, left hip A62, right knee A71, left knee A72, right foot A81, and left foot A82 as key points of person P10.

[0042] Furthermore, the analysis unit 11 sets up bones connecting these key points as a pseudo-skeletal structure of person P10, as shown below. Bone B1 connects head A1 and neck A2. Bone B21 connects neck A2 and right shoulder A31, and bone B22 connects neck A2 and left shoulder A32. Bone B31 connects right shoulder A31 and right elbow A41, and bone B32 connects left shoulder A32 and left elbow A42. Bone B41 connects right elbow A41 and right hand A51, and bone B42 connects left elbow A42 and left hand A52. Bone B51 connects neck A2 and right hip A61, and bone B52 connects neck A2 and left hip A62. Bone B61 connects the right hip A61 and the right knee A71, and bone B62 connects the left hip A62 and the left knee A72. Bone B71 connects the right knee A71 and the right foot A81, and bone B72 connects the left knee A72 and the left foot A82. When the analysis unit 11 generates skeletal data relating to the above-described skeletal structure, it supplies the generated skeletal data to the determination unit 12.

[0043] Next, an example of reference motion data will be explained with reference to Figure 5. Figure 5 is a diagram illustrating the reference motion data according to Embodiment 2. Table T10 shown in Figure 5 associates reference motion IDs (identification, identifiers) with skeletal data for multiple motion patterns. Reference motion ID (or motion ID) "R01" corresponds to the skeletal data for dance motion A. Similarly, the skeletal data corresponding to reference motion ID "R02" is dance motion B, and the skeletal data corresponding to reference motion ID "R03" is dance motion C, etc.

[0044] As described above, the reference motion data is stored with each motion pattern associated with a motion ID and related words. Each reference motion ID is linked to one or more skeletal data points.

[0045] Referring to Figure 6, the skeletal data related to the reference movement will be explained. Figure 6 is a diagram illustrating a first example of a reference movement according to Embodiment 2. Figure 6 shows the skeletal data of dance movement A, whose movement ID is "R01", among the reference movements included in the reference movement data. Figure 6 shows multiple skeletal data, including skeletal data F11 and skeletal data F12, arranged in the left-right direction. Skeletal data F11 is located to the left of skeletal data F12. Skeletal data F11 is a movement that captures the pattern changes of a person dancing a predetermined dance pattern over time.

[0046] The motion pattern with motion ID "R01" shown in Figure 6 means that the person assumes the pose corresponding to skeletal data F11, and then assumes the pose of skeletal data F12. Although two skeletal data sets have been explained here, the reference motion with motion ID "R01" may include skeletal data other than those mentioned above.

[0047] Figure 7 is a diagram illustrating a second example of a reference motion according to Embodiment 2. Figure 7 shows the skeletal data F31 related to the motion with motion ID "R03" shown in Figure 5. For the reference motion with motion ID "R03", one skeletal data F31 representing a static posture is registered as dance motion C.

[0048] As described above, the reference motion included in the reference motion data may include only one skeletal data or may include two or more skeletal data. The determination unit 12 compares the reference motion including the skeletal data described above with the skeletal data relating to the motion image received from the analysis unit 11 to determine whether or not there are similar reference motions. The determination unit 12 also generates data indicating similarity for motion images that are similar to the reference motion and supplies it to the labeling unit 13.

[0049] Next, the label data generated by the image processing system 20 will be explained with reference to Figure 8. Figure 8 is a diagram illustrating an example of label data. Figure 8 shows a time axis extending from left to right, strip-shaped label data formed along this time axis, and strip-shaped image data formed along the label data. In Figure 8, the label data and image data are shown as strips for ease of explanation. However, the strip-shaped label data and image data are each composed of data at a predetermined frame rate. Therefore, for example, 15 frames per second (15fps) image data is composed of data corresponding to one frame of image every 1 / 15th of a second. In this case, the label data is composed of data related to the label corresponding to each frame every 1 / 15th of a second.

[0050] The example shown in Figure 8 illustrates the state in which predetermined labels have been assigned to image data from time T10 to time T14. Specifically, the label data for the period from time T10 to time T11 is labeled "R04". Similarly, the label data for the period from time T11 to time T12 is labeled "R01", the operation ID is R02 from time T12 onwards is assigned, and the label for the period from time T13 to time T14 is labeled "R03".

[0051] As described above, the label data includes at least time-related data corresponding to the image data and data for the label to be assigned. The label data may be included in the image data or may be separate from the image data. The image processing system 20 generates label data corresponding to the image data, thereby enabling the image data to be segmented according to the reference operation. In other words, when a user of the information processing system 200 analyzes the movement of a person in the image data, the image data can be easily segmented using the label data.

[0052] Although Embodiment 2 has been described above, the image processing system 20 according to Embodiment 2 is not limited to the above configuration. For example, the labeling unit 13 may assign multiple types of labels to corresponding frames in the image data. This is the case, for example, when the determination unit 12 determines that one action image is related to multiple types of reference actions. With such a configuration, the image processing system 20 can divide the image data more flexibly.

[0053] Furthermore, the labeling unit 13 may assign labels in a manner that is editable by the user. User-editable manners include, for example, allowing the user to change the names of the labels included in the label data shown in Figure 8. User-editable manners also include, for example, allowing the user to adjust the time intervals that serve as boundaries between labels shown in Figure 8. User-editable manners also include, for example, merging adjacent labels or separating a single label using a predetermined time interval as a boundary. This allows the image processing system 20 to segment image data more efficiently.

[0054] As described above, Embodiment 2 provides an image processing system that can efficiently separate and process human movements from images captured by a camera.

[0055] <Embodiment 3> Next, Embodiment 3 will be described. Figure 9 is a block diagram showing the overall configuration of the image processing system according to Embodiment 3. In Embodiment 3, the information processing system and the image processing system are configured separately. Figure 9 shows that the image processing system 30 and the information processing system 210 are connected to each other via a network N1 so that they can communicate with each other. Figure 9 also shows that a camera 100, which is installed to capture the movements of a person, is connected to the network N1.

[0056] The image processing system 30 shown in Figure 9 will be described below. The image processing system 30 receives image data from the information processing system 210, generates labeled data by assigning labels to the received image data, and supplies the generated labeled data to the information processing system 210. The main components of the image processing system 30 are an analysis unit 11, a determination unit 12, a label assignment unit 13, a reference operation data storage unit 14, an image data acquisition unit 15, a selection unit 16, an output unit 17, and a storage unit 18.

[0057] In this embodiment, the analysis unit 11 includes a body analysis unit 111 and an upper limb analysis unit 112. The body analysis unit 111 recognizes body movement images from skeletal data relating to the structure of a person's body extracted from image data. In other words, the body analysis unit 111 has the same functions as the analysis unit 11 described in Embodiment 2.

[0058] The upper limb analysis unit 112 recognizes upper limb motion images from skeletal data of the upper limbs, including the movement of the person's fingers, extracted from image data. In other words, the upper limb analysis unit 112 extracts images of the person's upper limbs from image data, estimates a pseudo-skeleton of the upper limbs, including the movement of the person's fingers, from the extracted images of the upper limbs, and generates skeletal data corresponding to the estimated pseudo-skeleton.

[0059] In this embodiment, the determination unit 12 determines the relationship between the reference movements corresponding to the body movement images and upper limb movement images, respectively. The determination unit 12 may also determine, according to the user's settings, whether to include either one or both of the body movement images and upper limb movement images as subjects for analysis, and then make a determination on the movement images of the determined body parts.

[0060] The image data acquisition unit 15 acquires image data supplied from the information processing system 210. Upon receiving the image data, the image data acquisition unit 15 supplies the received image data to the analysis unit 11.

[0061] The selection unit 16 selects a reference action for labeling an image image obtained from reference action data related to multiple reference actions. In other words, the determination unit 12 in this embodiment makes a determination using the selected reference action. For example, if the motion image analyzed by the analysis unit 11 relates to body posture, the selection unit 16 selects a reference action related to body posture, and if the motion image analyzed by the analysis unit 11 relates to upper limb posture, the selection unit 16 selects a reference action related to upper limb posture. Alternatively, the selection unit 16 may select a reference action by receiving an operation from a user using the image processing system 30.

[0062] The output unit 17 outputs the label data generated by the labeling unit 13. In the example shown in Figure 9, the output unit 17 outputs the label data generated by the labeling unit 13 to the information processing system 210 via the network N1.

[0063] The storage unit 18 includes non-volatile memory and stores at least reference operation data. The storage unit 18 also stores image data acquired from the information processing system 210 and label data generated by the labeling unit 13. As a result, the image processing system 30 stores data in a state where the segmentation process of the image data received from the information processing system 210 has been completed, and can output it as a single set of data.

[0064] Next, the information processing system 210 will be described. The information processing system 210 has the function of analyzing motion images of a person contained in the image data received from the camera 100. The information processing system 210 may be, for example, a personal computer, a tablet PC, or a smartphone. The main components of the information processing system 210 are an image data acquisition unit 201, a motion analysis unit 202, an image data storage unit 203, an operation reception unit 204, and a display unit 205. Of these, the functions of the image data acquisition unit 201, the motion analysis unit 202, and the image data storage unit 203 are the same as those of the information processing system 200 in Embodiment 2.

[0065] The operation reception unit 204 is, for example, a keyboard that receives input from the user using the information processing system 210. Alternatively, the operation reception unit 204 may be superimposed on the display unit 205 and may be a touch panel configured to work in conjunction with the display unit 205. The display unit 205 includes a liquid crystal panel or organic electroluminescence, etc., and displays and presents image data and label data to the user.

[0066] Next, the skeletal data of the upper limbs will be explained with reference to Figure 10. Figure 10 is a diagram illustrating the skeletal data of the upper limbs. The image shown in Figure 10 is an upper limb image F40 extracted from an image 400 captured by camera 100 of person P10. In the image processing system 30, the upper limb analysis unit 112 extracts the body image F40 from the image captured by camera 100 and further sets the skeletal structure.

[0067] The upper limb analysis unit 112 extracts feature points from the image that could be key points of person P10, and detects key points from the extracted feature points. At this time, the upper limb analysis unit 112 extracts more key points, such as the head and fingers, than the analysis unit 11 shown in Figure 4. For example, the upper limb analysis unit 112 detects the right ear A11, left ear A12, right eye A13, and left eye A14 on person P10's head. The upper limb analysis unit 112 also detects, for example, the first joint A510 and the second joint A511 of the right thumb on person P10's right hand. Similarly, the upper limb analysis unit 112 detects, for example, the first joint A520 and the second joint A521 of the left thumb on person P10's left hand.

[0068] The upper limb analysis unit 112 detects key points of the fingers in this way. This allows the image processing system 30 to differentiate the movements of the person P10 based on the movement of the fingers.

[0069] Next, with reference to Figure 11, the processes performed by the image processing system 30 in this embodiment will be described. Figure 11 is a flowchart of the image processing method according to Embodiment 3.

[0070] The image processing system 30 has an image data acquisition unit 15 that acquires image data from the information processing system 210 (step S21). The image data acquisition unit 15 supplies the acquired image data to the analysis unit 11.

[0071] Next, the analysis unit 11 recognizes multiple motion images from the received image data (step S22). Here, the analysis unit 11 extracts images of a person from the image data. The analysis unit 11 recognizes whether the extracted images are full-body images of a person or images of the upper limbs. The analysis unit 11 then generates skeletal data for the recognized images.

[0072] Next, the selection unit 16 determines a reference operation for labeling from the skeletal data generated by the analysis unit 11 (step S23). Once the selection unit 16 has determined the reference operation, it notifies the determination unit 12 of the determined content. The content notified by the selection unit 16 is, for example, a signal indicating the skeletal data related to the reference operation.

[0073] Next, the determination unit 12, in accordance with the signal received from the selection unit 16, refers to the reference motion and determines whether or not the skeletal data generated by the analysis unit 11 from the motion image is related to the reference motion (step S24).

[0074] Next, the labeling unit 13 assigns labels to the image data according to the processing of the determination unit 12 (step S25). The labeling unit 13 stores the label data related to the labels assigned to the image data in the storage unit 18.

[0075] Next, the output unit 17 outputs the label data stored in the storage unit 18 to the information processing system 210 (step S26).

[0076] Next, the reference operation data in this embodiment will be described. Figure 12 is a diagram illustrating the reference operation data according to Embodiment 3. Table T20 shown in Figure 12 is the reference operation data relating to a predetermined work operation performed by a person. For example, registered operation ID "R11" is the skeletal data relating to the operation pattern of work operation A. Also, registered operation ID "R12" is the skeletal data relating to the operation pattern of work operation B.

[0077] Furthermore, Table T20 shown in Figure 12 differs from Table T10 shown in Figure 5 in that, among the reference operation data, the registered operation ID is "R00" and it includes "unregistered operations". When the image processing system 30 refers to the reference operations in Table T20, it associates "R00" with operation images that are not similar to the registered operation patterns. In other words, the labeling unit 13 can assign a label indicating that an operation is unregistered for operations that are not similar to any of the reference operations. This allows the user of the image processing system 30 to appropriately identify undefined operations included in operation images. With this configuration, the image processing system 30 can also perform separation processing on operation images that do not contain registered operation patterns.

[0078] Next, an example of an image showing label data to the user will be described with reference to Figure 13. Figure 13 is a diagram showing a first example of an image including label data according to Embodiment 3. The image 410 shown in Figure 13 includes an image F41 of a person P41 performing a predetermined task and label data D41 assigned by the image processing system 30. The label data D41 is a band-shaped display that extends horizontally across the screen. The label data D41 shows the passage of time from left to right, and as time passes, the labels assigned to the actions of person P41 are displayed as "R11", "R00", "R12", and "R16".

[0079] Below the label data D41, an indicator G41 showing the passage of time is displayed. The upper protrusion of indicator G41 moves horizontally across the screen, and its tip represents the passage of time for the label data. In addition, time passage information is displayed at the bottom of indicator G41. With the above display, image 410 shows the action image of person P41 performing the work action in correspondence with the label data D41. With this display, the image processing system 30 can present the action image and label data to the user in correspondence.

[0080] In this way, the labeling unit 13 assigns a label to frames of action images that are associated with a predetermined reference action, indicating that they are related to that reference action. On the other hand, the labeling unit 13 assigns a label to frames of action images that are not associated with any reference action, indicating that they are not related to any reference action. A user viewing image 410 can use this to analyze the actions performed by person P41. For example, the user can understand how long each action corresponding to a registered action ID is performed. The user can also understand the period of "R00" which does not correspond to a registered action ID, and further understand what person P41 is doing during that period.

[0081] Next, further examples of label data will be described with reference to Figure 14. Figure 14 is a diagram showing a second example of an image containing label data according to Embodiment 3. In image 420 shown in Figure 14, the label data differs from that in Figure 13. In image 420, the label data D42 shows, for example, "R12" and "R21" as the corresponding label data at the same time.

[0082] If the determination unit 12 indicates that one motion image is associated with multiple types of reference motions, the labeling unit 13 assigns multiple types of labels to the corresponding frames in the image data. Therefore, the label data D42 displays multiple corresponding labels during the period in which multiple labels are assigned. This allows the image processing system 30 to, for example, distinguish between the reference motions of the right fingers and the reference motions of the left fingers. Alternatively, the image processing system 30 can flexibly assign labels to motion images that are similar to multiple motion patterns.

[0083] Further examples of label data will be described with reference to Figure 15. Figure 15 is a diagram showing a third example of an image containing label data according to Embodiment 3. The image 430 shown in Figure 15 includes an image F42 of a person P42 performing a predetermined task, and an image F43 of a person P43 performing a predetermined task next to person P42. The image 430 also includes label data D42 corresponding to image F42 and label data D43 corresponding to image F43 as label data assigned by the image processing system 30.

[0084] In this way, when an image of motion includes multiple people, the image processing system 30 has an analysis unit 11 that extracts a body image or upper limb image for each of the multiple people and generates skeletal data for each extracted image. The determination unit 12 then determines the relationship between each extracted skeletal data and a reference motion, and the labeling unit 13 generates label data for each extracted skeletal data. This allows the user of the image processing system 30 to analyze the motions of multiple people while comparing them.

[0085] Embodiment 3 has been described above. Embodiment 3 provides an image processing system that can efficiently separate human movements from images captured by a camera.

[0086] <Example hardware configuration> The following describes how each functional configuration of the determination device in this disclosure is realized through a combination of hardware and software.

[0087] Figure 16 is a block diagram illustrating the hardware configuration of a computer. The determination device in this disclosure can realize the above-described functions using a computer 500 including the hardware configuration shown in the figure. The computer 500 may be a portable computer such as a smartphone or tablet terminal, or a stationary computer such as a PC. The computer 500 may be a dedicated computer designed to realize each device, or it may be a general-purpose computer. The computer 500 can realize the desired functions by installing a predetermined application.

[0088] Computer 500 has a bus 502, a processor 504, memory 506, a storage device 508, an input / output interface 510 (an interface is also called an I / F (Interface)), and a network interface 512. Bus 502 is a data transmission path for the processor 504, memory 506, storage device 508, input / output interface 510, and network interface 512 to send and receive data to and from each other. However, the method of connecting the processor 504 and the other components to each other is not limited to bus connection.

[0089] Processor 504 is a variety of processors such as a CPU, GPU, or FPGA. Memory 506 is main memory implemented using RAM (Random Access Memory), etc.

[0090] The storage device 508 is an auxiliary storage device implemented using a hard disk, SSD, memory card, or ROM (Read Only Memory). The storage device 508 stores a program for implementing a desired function. The processor 504 reads this program into memory 506 and executes it to implement each functional component of each device.

[0091] The input / output interface 510 is an interface for connecting the computer 500 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 510.

[0092] Network interface 512 is an interface for connecting computer 500 to a network.

[0093] The above describes examples of hardware configurations in this disclosure, but the embodiments described above are not limited thereto. This disclosure can also be implemented by having a processor execute a computer program to perform any processing.

[0094] In the examples described above, the program includes a set of instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrical, optical, acoustic or other forms of propagating signals.

[0095] Although embodiments have been described above, the configurations of the embodiments described above may be combined with each other, or some configurations may be replaced with other configurations. Furthermore, the configurations of the embodiments described above may be modified in various ways without departing from the spirit of the invention. In addition, the flowcharts used in the above description may be modified in the execution order of the processes performed in each embodiment, to the extent that it does not impede the function realized by the embodiment.

[0096] Although the present invention has been described above with reference to embodiments, the present invention is not limited thereto. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the invention.

[0097] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) An analysis means for recognizing multiple motion images that show a person's actions from image data of motion images spanning multiple consecutive frames of a person performing a series of actions, A determination means for determining whether the aforementioned motion image and a predetermined reference motion are related, The system includes labeling means for assigning labels to at least a portion of the consecutive frames of the motion image based on the determination. Image processing system. (Note 2) The analysis means recognizes the motion image from the skeletal data relating to the structure of the person's body extracted from the image data. The image processing system described in Appendix 1. (Note 3) The analysis means recognizes the motion image from skeletal data including the movement of a person's fingers extracted from the image data. The image processing system described in Appendix 1. (Note 4) The determination means makes the determination that the motion image is related to the reference motion when the skeletal data relating to the motion and the skeletal data as a reference motion are similar, based on the shapes of the elements constituting the skeletal data. The image processing system described in Appendix 2 or 3. (Note 5) The aforementioned analysis means is A body analysis means for recognizing body movement images from skeletal data relating to the structure of a person's body extracted from the aforementioned image data, The system includes an upper limb analysis means for recognizing upper limb motion images from skeletal data of the upper limbs, including the movement of a person's fingers, extracted from the aforementioned image data, The determination means performs the determination regarding the relationship between the reference movement corresponding to the body movement image and the upper limb movement image, respectively. The image processing system described in Appendix 1. (Note 6) The determination means determines, according to the user's settings, whether to include either or both of the body motion images and the upper limb motion images as subjects for analysis, and performs the determination based on the motion images of the determined body parts. The image processing system described in Appendix 5. (Note 7) The analysis means, if the image data includes multiple people, recognizes the motion image for each person. An image processing system as described in any one of the appendices 1 to 6. (Note 8) The labeling means assigns the label indicating that the frame relating to the motion image associated with a predetermined reference operation is related to the reference operation. An image processing system as described in any one of the appendices 1 to 7. (Note 9) The labeling means assigns the label indicating that the frame relating to the motion image is unrelated to any of the reference motions to the frame relating to the motion image which is unrelated to any of the reference motions. The image processing system described in Appendix 8. (Note 10) If the determination indicates that one of the motion images and multiple types of the reference motions are related, the labeling means assigns multiple types of labels to the corresponding frames in the image data. An image processing system as described in any one of the appendices 1 to 9. (Note 11) The labeling means assigns the label in a manner that is editable by the user. An image processing system as described in any one of the appendices 1 to 10. (Note 12) The system further comprises storage means for storing a plurality of predetermined reference operations in an updatable manner, The determination means performs the determination based on the updated reference operation. An image processing system as described in any one of the appendices 1 to 11. (Note 13) The system further comprises a selection means for selecting a reference operation from operation data relating to a plurality of reference operations to assign the label to the operation image, The determination means performs the determination based on the selected reference operation. An image processing system as described in any one of the appendices 1 to 11. (Note 14) Computers From image data of motion images spanning multiple consecutive frames of a person performing a series of actions, multiple motion images representing the person's actions are recognized. A determination is made as to whether the aforementioned motion image and a predetermined reference motion are related. Based on the above determination, labels are assigned to at least a portion of the consecutive frames in the motion image. Image processing methods. (Note 15) From image data of motion images spanning multiple consecutive frames of a person performing a series of actions, multiple motion images representing the person's actions are recognized. A determination is made as to whether the aforementioned motion image and a predetermined reference motion are related. Based on the above determination, labels are assigned to at least a portion of the consecutive frames in the motion image. A non-temporary, computer-readable medium containing a program that allows a computer to execute an image processing method. (Note 16) Image data acquisition means for acquiring image data of a person performing a series of actions, A determination means for determining whether the actions shown by each time-series interval in the aforementioned image data are similar to a predetermined reference action, A labeling means that assigns a label to the section in the image data indicating the type of operation represented by the section, based on the determination above. An image processing system equipped with the following features. [Explanation of symbols]

[0098] 10 Image Processing Systems 20 Image Processing Systems 30 Image Processing Systems 11 Analysis Department 12 Judgment section 13 Labeling section 14. Reference operation data storage unit 15 Image Data Acquisition Unit 16 Selection Section 17 Output section 18 Memory section 100 Cameras 111 Body Analysis Department 112 Upper limb analysis department 200 Information Processing Systems 201 Image Data Acquisition Unit 202 Motion analysis section 203 Image Data Storage Unit 204 Operation Reception Section 205 Display section 210 Information Processing Systems 500 Computers 504 Processors 506 memory 508 Storage Devices 510 Input / Output Interfaces 512 Network Interfaces N1 Network

Claims

1. An analysis means for recognizing multiple motion images showing a person's movements from multiple consecutive frames, A determination means for determining whether the aforementioned motion image and a predetermined reference motion are related, The system includes labeling means that, based on the determination, assigns a label to frames containing the motion image associated with the reference operation, and assigns a different label to frames containing the motion image not associated with any of the reference operations. The aforementioned analysis means is A body analysis means for recognizing a body movement image from skeletal data relating to the structure of a person's body extracted from the aforementioned frame, The system includes an upper limb analysis means for recognizing an upper limb motion image from skeletal data of the upper limbs, including the movement of a person's fingers, extracted from the aforementioned frame, The determination means performs the determination regarding the relationship between the reference movement corresponding to the body movement image and the upper limb movement image, respectively. Image processing system.

2. The analysis means recognizes the motion image from skeletal data relating to the structure of the person's body extracted from the frame. The image processing system according to claim 1.

3. The analysis means recognizes the motion image from skeletal data including the movement of the person's fingers extracted from the frame. The image processing system according to claim 1.

4. The determination means makes the determination that the motion image is related to the reference motion when the skeletal data relating to the motion and the skeletal data as a reference motion are similar, based on the shapes of the elements constituting the skeletal data. The image processing system according to claim 2 or 3.

5. The determination means determines, according to the user's settings, whether to include either or both of the body motion images and the upper limb motion images as subjects for analysis, and performs the determination based on the motion images of the determined body parts. The image processing system according to claim 1.

6. The analysis means, if the frame contains multiple people, recognizes the motion image for each person. The image processing system according to any one of claims 1 to 5.

7. The labeling means assigns the label indicating that the frame relating to the motion image associated with a predetermined reference operation is related to the reference operation. The image processing system according to any one of claims 1 to 6.

8. Computers Recognizing multiple motion images showing a person's movements from multiple consecutive frames, A determination is made as to whether the aforementioned motion image and a predetermined reference motion are related. Based on the above determination, a label is assigned to the frame containing the motion image associated with the reference operation, and a different label is assigned to the frame containing the motion image not associated with any of the reference operations. Recognizing the aforementioned multiple motion images means From the skeletal data relating to the structure of the person's body extracted from the aforementioned frame, a body movement image is recognized. This includes recognizing an upper limb motion image from skeletal data of the upper limbs, including the movement of a person's fingers, extracted from the aforementioned frame, The determination includes making a determination regarding the relationship between the reference movement corresponding to the body movement image and the upper limb movement image, respectively. Image processing methods.

9. Recognizing multiple motion images showing a person's movements from multiple consecutive frames, A determination is made as to whether the aforementioned motion image and a predetermined reference motion are related. Based on the above determination, a label is assigned to the frame containing the motion image associated with the reference operation, and a different label is assigned to the frame containing the motion image not associated with any of the reference operations. The image processing method is executed by a computer. Recognizing the aforementioned multiple motion images means From the skeletal data relating to the structure of the person's body extracted from the aforementioned frame, a body movement image is recognized. This includes recognizing an upper limb motion image from skeletal data of the upper limbs, including the movement of a person's fingers, extracted from the aforementioned frame, The determination includes making a determination regarding the relationship between the reference movement corresponding to the body movement image and the upper limb movement image, respectively. program.