Trained model generation device, trained model generation method, orientation determination device, and computer-readable recording medium

Abstract

A trained model generation device 10 comprises a training data acquisition unit 11 that acquires training data including skeleton information and image data of a person, an image feature amount extraction unit 12 that extracts an image feature amount from the image data by using a first machine learning mode, a skeleton feature amount extraction unit 13 that extracts a skeleton feature amount from the skeleton information by using a second machine learning model, a similarity degree calculation unit 14 that calculates the degree of similarity between the image feature amount and the skeleton feature amount for each combination of the image data and the skeleton information, a corresponding skeleton information acquisition unit 15 for acquiring corresponding skeleton information corresponding to the image data, a skeleton similarity degree calculation unit 16 for calculating the degree of similarity in skeleton between the corresponding skeleton information and the skeleton information for each combination, and a parameter update unit 17 for updating the parameters of the first machine learning model and the second machine learning model by using the difference between the similarity degree and the skeleton similarity degree for each combination.

Description

Learning model generation device, learning model generation method, posture determination device, and computer-readable recording medium

[0001] The present disclosure relates to a learning model generation device and a learning model generation method for generating a learning model for determining a person's posture from an image, and further to a computer-readable recording medium storing a program for realizing these. The present disclosure also relates to a posture determination device for determining a person's posture from an image.

[0002] In recent years, a multimodal model based on language and images called CLIP has been attracting attention in the field of machine learning (see, for example, Non-Patent Document 1). CLIP can use image data and explanatory text instead of image data and labels as training data, and can learn co-occurrence relationships between images and text. Therefore, CLIP can increase the degree of freedom of training data in neural network machine learning.

[0003] Here, machine learning in CLIP will be described with reference to FIG. 13 . FIG. 13 is an explanatory diagram illustrating machine learning by CLIP. As shown in FIG. 13 , a combination of image data and corresponding text data is used as training data. CLIP also uses an image encoder that outputs feature values ​​when image data is input, and a text encoder that outputs feature values ​​when text data is input. Both the image encoder and the text encoder are constructed using a model that has learned the relationship between input data and feature values ​​through machine learning.

[0004] Then, the image encoder calculates the feature quantity I １ ~I Ｎ The text encoder outputs the feature T １ ~T Ｎ Furthermore, for all combinations of each image data and each text data, the similarity between the feature amount of the image data and the feature amount of the text data is calculated.

[0005] Furthermore, identity determination is performed for each combination of image data and text data. The matrix in Fig. 13 shows the identity determination results. In the example of Fig. 13, if the image and text correspond, the result is "1" (positive), and if the image and text do not correspond, the result is "0" (negative).

[0006] Then, for each combination of image data and text data, the difference between the feature similarity and a value indicating the identity determination result (1 or 0 in the example of FIG. 13) is calculated, and the parameters of the machine learning models in the image encoder and the text encoder are updated based on the calculation result. That is, in FIG. 13, each parameter is updated so that the feature similarity between corresponding image data and text data becomes "1". On the other hand, in FIG. 13, each parameter is updated so that the feature similarity between non-corresponding image data and text data becomes "0". Machine learning using CLIP is thought to be applicable to other classifiers as well.

[0007] Alec Radford, 11 others, “Learning Transferable Visual Models From Natural Language Supervision”, arXiv:2103.00020v1 [cs.CV], February 26, 2021

[0008] Recently, neural networks have been used to determine human posture from image data for the purposes of analyzing work conditions in factories and health checkups. It is expected that the aforementioned CLIP will be introduced into the neural networks used for such determinations in order to increase the flexibility of training data.

[0009] However, when determining a person's posture, image data and skeletal information of the person are used as training data, and in this case, if identity is determined using "1" or "0" as in the example shown in Figure 13, the accuracy of the determination may be significantly reduced.

[0010] An example of an objective of the present disclosure is to improve the accuracy of human posture determination using a machine learning model while increasing the degree of freedom of training data.

[0011] In order to achieve the above object, a learning model generation device according to one aspect of the present disclosure comprises: a training data acquisition unit that acquires training data including image data of a person and skeletal information of the person in the image data; an image feature extraction unit that extracts features of the image data from the image data using a first machine learning model; a skeletal feature extraction unit that extracts features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation unit that sets combinations of the image data and the skeletal information and calculates a similarity between the features of the image data and the features of the skeletal information for each set combination; a corresponding skeletal information acquisition unit that acquires corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation unit that calculates a similarity between the corresponding skeletal information and the skeletal information as a skeletal similarity for each set combination; and a parameter update unit that calculates a difference between the calculated similarity and the skeletal similarity for each set combination, and updates parameters of the first machine learning model and the second machine learning model using the calculated difference.

[0012] In order to achieve the above object, a posture determination device according to one aspect of the present disclosure is characterized by comprising: an image feature extraction unit that extracts image features of image data of a person to be subjected to posture determination using a first machine learning model that has learned the relationship between image data and its features through machine learning; a skeletal feature extraction unit that extracts skeletal features of each piece of skeletal information for each of a plurality of predefined postures using a second machine learning model that has learned the relationship between a person's skeletal information and its features through machine learning; a similarity calculation unit that calculates the similarity between the extracted image features and each of the extracted skeletal features; and a posture determination unit that identifies the skeletal feature with the highest similarity, further identifies a posture corresponding to the identified skeletal feature, and determines the identified posture as the posture of the person to be subjected to posture determination.

[0013] Furthermore, to achieve the above object, a learning model generation method according to one aspect of the present disclosure is characterized by comprising: a training data acquisition step of acquiring training data including image data of a person and skeletal information of the person in the image data; an image feature extraction step of extracting features of the image data from the image data using a first machine learning model; a skeletal feature extraction step of extracting features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation step of setting combinations of the image data and the skeletal information and calculating a similarity between the features of the image data and the features of the skeletal information for each set combination; a corresponding skeletal information acquisition step of acquiring corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation step of calculating a similarity between the corresponding skeletal information and the skeletal information as a skeletal similarity for each set combination; and a parameter update step of calculating a difference between the calculated similarity and the skeletal similarity for each set combination, and updating parameters of the first machine learning model and the second machine learning model using the calculated difference.

[0014] Furthermore, to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure is characterized in that it records a program including instructions for causing a computer to execute the following: a training data acquisition step of acquiring training data including image data of a person and skeletal information of the person in the image data; an image feature extraction step of extracting features of the image data from the image data using a first machine learning model; a skeletal feature extraction step of extracting features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation step of setting combinations of the image data and the skeletal information and calculating, for each set combination, a similarity between the features of the image data and the features of the skeletal information; a corresponding skeletal information acquisition step of acquiring corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation step of calculating, for each set combination, the similarity between the corresponding skeletal information and the skeletal information as skeletal similarity; and a parameter update step of calculating, for each set combination, a difference between the calculated similarity and the skeletal similarity, and updating parameters of the first machine learning model and the second machine learning model using the calculated difference.

[0015] As described above, according to the present disclosure, in determining a person's posture using a machine learning model, it is possible to improve the accuracy of determination while increasing the degree of freedom of training data.

[0016] FIG. 1 is a diagram illustrating a schematic configuration of an example of a learning model generation device. FIG. 2 is a diagram specifically illustrating the configuration of an example of a learning model generation device. FIG. 3 is a diagram illustrating an example of image data and skeletal information used as training data. FIG. 4 is a diagram illustrating an example of a bone length vector and a bone length ratio vector. FIG. 5 is a diagram illustrating an example of an angle vector obtained by concatenating a camera vector and a physique vector. FIG. 6 is a diagram schematically illustrating an example of a parameter update process in the learning model generation device. FIG. 7 is a flow chart illustrating an example of the operation of the learning model generation device. FIG. 8 is a diagram illustrating a schematic configuration of an example of a posture determination device. FIG. 9 is a diagram specifically illustrating the configuration of an example of a posture determination device. FIG. 10 is a diagram illustrating an example of a posture determination process. FIG. 11 is a flow chart illustrating an example of the operation of the posture determination device. FIG. 12 is a block diagram illustrating an example of a computer that realizes the learning model generation device and the posture determination device. FIG. 13 is an explanatory diagram illustrating machine learning using CLIP.

[0017] First Embodiment First, in a first embodiment, a learning model generating device, a learning model generating method, and a program will be described with reference to FIGS.

[0018] [Device Configuration] First, a schematic configuration of an example of a learning model generation device will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of an example of a learning model generation device.

[0019] The learning model generation device 10 shown in Fig. 1 is a device that generates a learning model that determines a person's posture from an image. As shown in Fig. 1, the learning model generation device 10 includes a training data acquisition unit 11, an image feature extraction unit 12, a skeleton feature extraction unit 13, a similarity calculation unit 14, a corresponding skeleton information acquisition unit 15, a skeleton similarity calculation unit 16, and a parameter update unit 17.

[0020] The training data acquisition unit 11 acquires training data. The training data includes image data of a person and skeletal information of the person in the image data. The image feature extraction unit 12 extracts image data features (hereinafter referred to as "image features") from the acquired image data using a first machine learning model. The skeletal feature extraction unit 13 extracts skeletal information features (hereinafter referred to as "skeletal features") from the skeletal information using a second machine learning model.

[0021] The similarity calculation unit 14 sets combinations of image data and skeletal information, and calculates the similarity between the image feature amount and the skeletal feature amount for each set combination.

[0022] The corresponding skeletal information acquisition unit 15 acquires corresponding skeletal information corresponding to people in the image data. The skeletal similarity calculation unit 16 calculates the similarity between the corresponding skeletal information and the skeletal information for each set combination as skeletal similarity.

[0023] The parameter update unit 17 calculates, for each set combination, the difference between the similarity calculated by the similarity calculation unit 14 and the skeletal similarity calculated by the skeletal similarity calculation unit 16. Furthermore, the parameter update unit 17 uses the calculated difference to update the parameters of the first machine learning model and the parameters of the second machine learning model.

[0024] In this way, the learning model generation device 10 can use image data and skeletal information, rather than image data and labels, as training data, and can therefore determine a person's posture using the generated learning model. Furthermore, the identity between the image data and the skeletal information is determined by calculating the similarity between the skeletal information corresponding to the image data, and the result of the identity determination is not a binary value of 0 or 1. Therefore, the learning model generation device 10 can increase the flexibility of the training data while improving the accuracy of the determination of a person's posture using a machine learning model.

[0025] Next, the configuration and functions of the learning model generation device 10 will be specifically described with reference to Figures 2 to 6. Figure 2 is a block diagram specifically showing the configuration of an example of the learning model generation device.

[0026] 2, the learning model generation device 10 is connected to a database 30 so as to be able to communicate data with the database 30. The database 30 stores image data of people and skeletal information of people in the image data, which serve as training data.

[0027] The training data acquisition unit 11 acquires training data from the database 30. Then, the training data acquisition unit 11 inputs the image data acquired as training data to the image feature extraction unit 12, and inputs the skeletal information acquired as training data to the skeletal feature extraction unit 13.

[0028] FIG. 3 is a diagram showing an example of image data and skeletal information used as training data. In the example of FIG. 3, arrows indicate corresponding image data and skeletal information. The image data is data obtained by photographing a person with a camera and is output from the camera. The image data may be image data of a still image or image data of each frame constituting a video. In the example of FIG. 3, the skeletal information is shown as a skeleton, but in reality, it is composed of the coordinates of each joint of the person in the image data. The coordinates may be two-dimensional or three-dimensional. The origin of the coordinates of each joint is set, for example, based on the camera. Furthermore, the skeletal information may include various information such as information representing body type (e.g., parameters indicating the degree of obesity).

[0029] In the first embodiment, when image data is input, the image feature extraction unit 12 extracts image features using a first machine learning model 21 and outputs the extracted image features. The first machine learning model 21 is a machine learning model that has previously learned the relationship between image data and its features. An example of the first machine learning model 21 is a neural network.

[0030] In the first embodiment, when skeleton information is input, the skeleton feature extraction unit 13 extracts skeleton features using the second machine learning model 22 and outputs the extracted skeleton features. The second machine learning model 22 is a machine learning model that has previously learned the relationship between the skeleton information and its features. Another example of the second machine learning model 22 is a neural network.

[0031] The first machine learning model 21 and the second machine learning model 22 are actually implemented by a machine learning program executed on a computer. The first machine learning model 21 and the second machine learning model 22 may also be implemented in a device (computer) separate from the learning model generation device 10.

[0032] Here, the output skeletal feature is "T ｉ " and the output image feature is expressed as "I ｊ ". Both i and j are integers from 1 to N (i, j = 1, ..., N), and are numbers assigned to the image data and skeletal information that serve as training data. The value of N matches the number of image data and skeletal information that serve as training data. Furthermore, image data and skeletal information that are assigned the same number correspond to each other. In other words, for example, the skeleton represented by the first skeletal information corresponds to the skeleton of the person appearing in the first image data.

[0033] Furthermore, the image data and skeletal information corresponding to each other are linked and stored in the database 30. The linking is performed by the meta information of the image data and the skeletal information stored in the database 30. Specifically, the meta information of the image data includes an identifier of the corresponding skeletal information, and the meta information of the skeletal information includes an identifier of the corresponding image data.

[0034] In the first embodiment, the similarity calculation unit 14 calculates the skeletal feature amount T ｉ and image feature I ｊ Similarity sim(T ｉ , I ｊ Specifically, the similarity calculation unit 14 calculates the skeletal feature T ｉ and image feature I ｊ and the similarity sim(T ｉ , I ｊ ) and calculate the cosine similarity or Euclidean distance.

[0035] In this embodiment, the corresponding skeleton information acquisition unit 15 first accesses the database 30 to acquire meta information for each image data item and meta information for each skeleton information item. Next, the corresponding skeleton information acquisition unit 15 calculates the similarity sim(T ｉ , I ｊ The corresponding skeletal information acquisition unit 15 then acquires the identified corresponding skeletal information from the database 30.

[0036] The skeleton similarity calculation unit 16 calculates the skeleton similarity S between the i-th corresponding skeleton information and the j-th skeleton information for each combination set by the similarity calculation unit 14. ｉ，ｊ Specifically, the skeleton similarity calculation unit 16 calculates the skeleton similarity S ｉ，ｊ For example, the cosine similarity between the coordinate values ​​of the joint points or the cosine similarity between the angle vectors is calculated as follows. The angle vector will be described later.

[0037] Here, the skeletal similarity S ｉ，ｊ The calculation method of the skeleton similarity S ｉ，ｊ When the cosine similarity between the coordinate values ​​of the joint points is calculated, the skeletal similarity calculation unit 16 calculates the skeletal similarity S ｉ，ｊ In the following formula 1, P ｉ indicates a coordinate vector obtained from the coordinate values ​​of each joint point included in the i-th corresponding skeleton information. ｊ indicates a coordinate value vector obtained from the coordinate values ​​of each joint point included in the j-th skeleton information.

[0038]

[0039] In addition, the skeletal similarity S ｉ，ｊWhen the cosine similarity between angle vectors is calculated as follows, for example, the following process is executed. First, the skeletal similarity calculation unit 16 calculates a camera posture vector for each piece of image data that has become training data. Examples of camera posture vectors include a vector indicating the angle between the optical axis direction of the camera and the vertical direction, and a vector indicating the angle between the optical axis of the camera and a part of a person. Note that the camera posture vector may be calculated in advance for each piece of image data.

[0040] Next, the skeleton similarity calculation unit 16 calculates the average vector of the camera posture vectors for each image data. mean Furthermore, the skeleton similarity calculation unit 16 calculates the vector of the camera used to take the image (hereinafter referred to as "camera vector") for each piece of image data. For example, the camera vector of person A is expressed as "cam A ”, and the camera vector of person B is “cam B ", the camera vector is calculated by the following equation 2.

[0041]

[0042] Next, the skeletal similarity calculation unit 16 calculates a bone length vector for each piece of skeletal information using the coordinates of each joint contained in it, and further calculates a "bone length ratio vector" from the calculated bone length vector.

[0043] Fig. 4 shows an example of a bone length vector and a bone length ratio vector. As shown in Fig. 5, the bone length vector is composed of "the length from the right shoulder to the right elbow," "the length from the right elbow to the right wrist," "the length from the right hip to the right ankle," "the length from the left hip to the left ankle," etc. Also, as shown in Fig. 4, each length is calculated from the difference in coordinate values ​​(three-dimensional coordinates) between joints. The bone length ratio vector is calculated by dividing each length constituting the bone length vector by a reference length.

[0044] The skeletal similarity calculation unit 16 also calculates an average vector of the ratio vectors of the bone lengths of all the people whose skeletal information is the target. meanFurthermore, the skeleton similarity calculation unit 16 calculates a physique vector representing the physique of the target person using the average vector for each skeleton information. For example, the physique vector of person A is expressed as "phy A ", and the physique vector of person B is "phy B ", the physique vector is calculated by the following equation 3.

[0045]

[0046] Next, the skeleton similarity calculation unit 16 concatenates the camera vector and the physique vector for each piece of skeleton information, as shown in Fig. 6. The vector obtained by concatenation is the above-mentioned "angle vector." Fig. 5 is a diagram showing an example of an angle vector obtained by concatenating the camera vector and the physique vector.

[0047] Thereafter, for each combination set by the similarity calculation unit 14, the corresponding skeleton information I ｉ The angle vector obtained for ｊ The similarity between the angle vector obtained for and is calculated. The similarity is calculated by using the cosine similarity (cos_sim(cam i +phy i , cam j +phy j The calculated similarity is called the skeletal similarity S ｉ，ｊ This becomes:

[0048] Furthermore, the angle vector may be a vector indicating the angle between the bone connecting the joints and the optical axis of the camera. In this case, the skeletal similarity calculation unit 16 first calculates the vector b ｋ，ｌ Specifically, the vector b ｋ，ｌ is expressed as the difference in three-dimensional coordinate values ​​for each combination of two joint points (k, l), as shown in the following equation 4.

[0049]

[0050] The combination of joint points (k, l) is determined in advance. The combination of joint points (k, l) may be determined as a natural combination that represents the skeleton of a person, or may be a combination of joint points selected randomly.

[0051] Next, the skeletal similarity calculation unit 16 acquires a vector C indicating the optical axis direction of the camera. It is assumed that the vector C has been measured in advance. Note that when the coordinate value vector P of the joint point is expressed in the camera coordinate system, it is expressed as C=(0,0,1).

[0052] Furthermore, the skeletal similarity calculation unit 16 calculates a vector C indicating the optical axis direction of the camera and a vector b representing the bones connecting the joint points using the following equation 5. ｋ，ｌ The angle θ ｋ，ｌ Calculate.

[0053]

[0054] The skeletal similarity calculation unit 16 calculates the above-mentioned vector b ｋ，ｌ Calculation, acquisition of vector C indicating the optical axis direction of the camera, and angle θ ｋ，ｌ Then, the skeleton similarity calculation unit 16 calculates the obtained angle θ ｋ，ｌ Arrange them in order to create the vector Θ.

[0055]

[0056] Furthermore, the skeletal similarity calculation unit 16 performs the above-described process on all the training data to create the vector Θ.

[0057] Next, the similarity calculation unit 14 calculates the similarity between the vectors Θ for each combination of image data and skeleton information as a skeleton similarity S ｉ，ｊ It is calculated as follows.

[0058]

[0059] The parameter update unit 17 calculates the similarity sim(T ｉ , I ｊ ) and the skeletal similarity S ｉ，ｊ and updates the parameters of the first machine learning model 21 and the second machine learning model 22 using the calculated difference. ｉ，ｊis normalized to define the range of its values ​​as the similarity sim(T ｉ , I ｊ ) value range.

[0060] Parameter updating will be specifically described with reference to Fig. 6. Fig. 6 is a diagram schematically illustrating an example of parameter updating processing in the learning model generation device. In Fig. 6, the numerical values ​​shown in the matrix are the skeletal similarity S calculated by the skeletal similarity calculation unit 16. ｉ，ｊ The parameter update unit 17 updates the skeletal feature T ｉ and image feature I ｊ Similarity sim(T ｉ , I ｊ The parameters of the first machine learning model 21 and the second machine learning model 22 are updated so that the values ​​of the first machine learning model 21 and the second machine learning model 22 are the corresponding values ​​on the matrix.

[0061] [Device Operation] Next, the operation of the learning model generation device 10 will be described with reference to FIG. 7. FIG. 7 is a flow diagram showing an example of the operation of the learning model generation device. In the following description, reference will be made to FIGS. 1 to 6 as appropriate. In addition, in the first embodiment, the learning model generation method is implemented by operating the learning model generation device. Therefore, the description of the learning model generation method in the first embodiment will be replaced by the following description of the operation of the learning model generation device 10.

[0062] 7 , first, the training data acquisition unit 11 acquires training data from the database 30 (step A1). Then, the training data acquisition unit 11 inputs the image data acquired as the training data to the image feature extraction unit 12, and inputs the skeletal information acquired as the training data to the skeletal feature extraction unit 13.

[0063] Next, when the skeleton information is input, the skeleton feature extraction unit 13 extracts skeleton features using the second machine learning model 22 (step A2). Furthermore, the skeleton feature extraction unit 13 outputs the extracted skeleton features to the similarity calculation unit 14.

[0064] Next, when the image data is input, the image feature extraction unit 12 extracts image features using the first machine learning model 21 (step A3). The image feature extraction unit 12 also outputs the extracted image features to the similarity calculation unit 14.

[0065] Next, the similarity calculation unit 14 calculates the skeletal feature value T extracted in step A2 for each combination of image data and skeletal information. ｉ and the image feature I extracted in step A3 ｊ Similarity sim(T ｉ , I ｊ ) is calculated (step A4).

[0066] Next, for each of the image data acquired in step A1, the corresponding skeletal information acquisition unit 15 acquires, from the database 30, skeletal information corresponding to the person in the image data (corresponding skeletal information) (step A5).

[0067] Specifically, in step A5, the corresponding skeletal information acquisition unit 15 accesses the database 30 to acquire meta information for each image data and meta information for each skeletal information. Then, the corresponding skeletal information acquisition unit 15 calculates the similarity sim(T ｉ , I ｊ ) is calculated, the skeleton information (corresponding skeleton information) corresponding to the image data is identified, and the identified corresponding skeleton information is obtained from the database 30.

[0068] Next, the skeleton similarity calculation unit 16 calculates the skeleton similarity S between the corresponding skeleton information and the skeleton information for each combination of image data and skeleton information. ｉ，ｊ is calculated (step A6).

[0069] Next, the parameter update unit 17 calculates the similarity sim(T ｉ , I ｊ ) and the skeletal similarity S ｉ，ｊ Then, the parameter update unit 17 uses the difference calculated in step A7 to update the parameters of the first machine learning model 21 and the second machine learning model 22 (step A8).

[0070] As described above, according to the first embodiment, it is possible to use image data and skeletal information as training data to update the parameters of the first machine learning model 21 and the second machine learning model 22. Therefore, by using the first machine learning model 21 and the second machine learning model 22, it is possible to determine a person's posture from image data.

[0071] Furthermore, the parameters of the first machine learning model 21 and the second machine learning model 22 are the skeletal feature T ｉ and image feature I ｊ Similarity sim(T ｉ , I ｊ ) is the corresponding skeleton similarity S ｉ，ｊ In other words, the parameters are not updated using binary values ​​as in the past. Therefore, in determining a person's posture using a machine learning model, it is possible to increase the degree of freedom of the training data and improve the determination accuracy.

[0072] [Program] In the first embodiment, an example of the program is a program that causes a computer to execute steps A1 to A8 shown in FIG. 7. By installing and executing this program on a computer, the learning model generation device 10 and the learning model generation method can be realized. In this case, the processor of the computer functions and performs processing as a training data acquisition unit 11, an image feature extraction unit 12, a skeleton feature extraction unit 13, a similarity calculation unit 14, a corresponding skeleton information acquisition unit 15, a skeleton similarity calculation unit 16, and a parameter update unit 17. Examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0073] In addition, in the first embodiment, the program may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the training data acquisition unit 11, the image feature extraction unit 12, the skeletal feature extraction unit 13, the similarity calculation unit 14, the corresponding skeletal information acquisition unit 15, the skeletal similarity calculation unit 16, and the parameter update unit 17.

[0074] Second Embodiment Next, in a second embodiment, a posture determination device, a posture determination method, and a program will be described with reference to FIGS.

[0075] [Device Configuration] First, a schematic configuration of an example of a posture determination device will be described with reference to Fig. 8. Fig. 8 is a diagram showing a schematic configuration of an example of a posture determination device.

[0076] 8 is a device for determining the posture of a person in image data. As shown in FIG. 1, posture determination device 40 includes an image feature extraction unit 41, a skeletal feature extraction unit 42, a similarity calculation unit 43, and a posture determination unit 44.

[0077] The image feature extraction unit 41 extracts image feature values ​​from image data of a person whose posture is to be determined, using a first machine learning model that has learned the relationship between image data and its feature values ​​through machine learning. The skeletal feature extraction unit 42 extracts skeletal feature values ​​of the skeletal information for each of a plurality of predefined postures, using a second machine learning model that has learned the relationship between human skeletal information and its feature values ​​through machine learning.

[0078] The similarity calculation unit 43 calculates the similarity between the extracted image feature and each of the extracted skeletal feature. The posture determination unit 44 identifies the skeletal feature that has the highest similarity, and further identifies a posture corresponding to the identified skeletal feature, and determines the identified posture as the posture of the person whose posture is to be determined.

[0079] As described above, posture determination device 40 includes image feature extraction unit 41 and skeletal feature extraction unit 42, and can extract features from both image data and skeletal information. Therefore, posture determination device 40 can determine the posture of a person in image data by calculating the similarity between image feature values ​​and skeletal feature values.

[0080] Next, the configuration and functions of the posture determination device 40 will be specifically described with reference to Fig. 9. Fig. 9 is a configuration diagram specifically showing the configuration of an example of the posture determination device. Fig. 10 is a diagram showing an example of posture determination processing.

[0081] As shown in FIG. 9 , the posture determination device 40 includes an input receiving unit 45 and a memory unit 46 in addition to the image feature extraction unit 41, skeletal feature extraction unit 42, similarity calculation unit 43, and posture determination unit 44 described above.

[0082] The input accepting unit 41 accepts input of skeletal information for a plurality of predefined postures (standing, spreading arms, crouching, etc.). Furthermore, the input accepting unit 41 also accepts input of image data of a person whose posture is to be determined. The input accepting unit 41 also stores the accepted input skeletal information and image data in the storage unit 46. As described in the first embodiment, the skeletal information is information composed of the coordinates of each joint of the person.

[0083] The image feature extraction unit 41 acquires image data from the storage unit 46, inputs it to the first machine learning model 51 to extract image features, and outputs the extracted image features to the similarity calculation unit 43. The skeletal feature extraction unit 42 acquires each piece of skeletal information from the storage unit 46, inputs the acquired skeletal information to the second machine learning model 52 to extract each skeletal feature, and outputs the extracted skeletal feature to the similarity calculation unit 43.

[0084] The first machine learning model 51 and the second machine learning model 52 are actually implemented by a machine learning program executed on a computer. The first machine learning model 51 and the second machine learning model 52 may also be implemented in a device (computer) separate from the posture determination device 40.

[0085] Furthermore, the parameters of the first machine learning model 51 and the second machine learning model 52 are updated according to the following steps (a) to (c): (a) For each combination of sample image data and sample skeletal information, the similarity between the feature amounts of the sample image data and the feature amounts of the sample skeletal information is calculated; (b) For each combination of sample image data and sample skeletal information, the similarity between the corresponding skeletal information corresponding to the person in the sample image data and the skeletal information is calculated as skeletal similarity; (c) For each combination of sample image data and sample skeletal information, the difference between the similarity calculated in (a) and the skeletal similarity calculated in (b) is calculated, and the calculated difference is used to update the parameters of the first machine learning model 51 and the second machine learning model 52.

[0086] Specifically, the first machine learning model 51 and the second machine learning model 52 are generated by the learning model generation device 10 described in embodiment 1. That is, the first machine learning model 51 is the first machine learning model 21 whose parameters have been updated by the learning model generation device 10. The second machine learning model 52 is the second machine learning model 22 whose parameters have been updated by the learning model generation device 10.

[0087] The similarity calculation unit 43 calculates the similarity between each of the image feature and the skeletal feature. Specifically, the similarity calculation unit 43 calculates the cosine similarity or the Euclidean distance as the similarity, similar to the similarity calculation unit 14 described in the first embodiment.

[0088] Furthermore, in the parameter update of the machine learning model described in the first embodiment, in the case of a combination of linked image data and skeletal information, the similarity sim(T ｉ , I ｊ ) becomes the maximum value. Therefore, as described above, the posture determination unit 44 identifies the skeletal feature amount that results in the highest similarity.

[0089] For example, in the example of FIG. 10, the number of defined postures is N. Therefore, the similarity calculation unit 43 calculates the image feature quantity I １ and the skeletal feature T １ To~T Ｎ The similarity sim(T１ , I １ ) ~ (T ５ , I Ｎ In this case, the similarity sim(T ３ , I １ ) is the largest. Therefore, the posture determination unit 44 determines whether the skeletal feature T ３ The posture corresponding to the position is identified, and the identified posture is determined to be the posture of the person whose posture is to be determined.

[0090] [Device Operation] Next, the operation of the posture determination device 40 will be described with reference to FIG. 11. FIG. 11 is a flow diagram showing an example of the operation of the posture determination device. In the following description, reference will be made to FIGS. 8 to 10 as appropriate. In addition, in the second embodiment, the posture determination method is implemented by operating the posture determination device. Therefore, the description of the posture determination method in the second embodiment will be replaced by the following description of the operation of the posture determination device 40.

[0091] 11 , first, the input receiving unit 41 receives input of skeletal information for a plurality of predefined postures (step B1). The input receiving unit 41 stores the received input skeletal information in the storage unit 46.

[0092] Next, the input receiving unit 41 receives input of image data of the person whose posture is to be determined (step B2), and stores the received image data in the storage unit 46.

[0093] Next, the image feature extraction unit 41 acquires the image data from the storage unit 46, inputs it to the first image model 51, and extracts image features (step B3). The image feature extraction unit 41 also outputs the extracted image features to the similarity calculation unit 43.

[0094] Next, the skeletal feature extraction unit 42 acquires each piece of skeletal information from the storage unit 46 and inputs each piece of acquired skeletal information to the second machine learning model 52 to extract skeletal features (step B4). The skeletal feature extraction unit 42 also outputs each extracted skeletal feature to the similarity calculation unit 43.

[0095] Next, the similarity calculation unit 43 calculates the similarity between the image feature extracted in step B3 and each of the skeletal feature extracted in step B4 (step B5).

[0096] Next, the posture determination unit 44 identifies the skeletal feature with the highest similarity based on the similarity calculated in step B5 (step B6). Furthermore, the posture determination unit 44 identifies a posture corresponding to the skeletal feature identified in step B6, and determines the identified posture as the posture of the person to be subjected to posture determination (step B7).

[0097] As described above, according to the second embodiment, it is possible to determine the posture of a person in image data. Furthermore, since the determination is performed using image features and skeletal features, the accuracy of the determination is improved.

[0098] [Program] In the second embodiment, an example of the program is a program that causes a computer to execute steps B1 to B7 shown in Fig. 11. By installing and executing this program on a computer, it is possible to realize posture determination device 40 and posture determination method. In this case, the processor of the computer functions as image feature extraction unit 41, skeletal feature extraction unit 42, similarity calculation unit 43, posture determination unit 44, and input reception unit 45 and performs processing.

[0099] The storage unit 46 may be implemented by a storage device such as a hard disk provided in the computer, or by a storage device of another computer. Examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0100] In addition, in the second embodiment, the program may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the image feature extraction unit 41, skeletal feature extraction unit 42, similarity calculation unit 43, posture determination unit 44, and input reception unit 45.

[0101] [Physical Configuration] Here, a computer that realizes the learning model generation device and the posture determination device by executing the program in embodiment 1 or 2 will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of a computer that realizes the learning model generation device and the posture determination device.

[0102] 12, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0103] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0104] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0105] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0106] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0107] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0108] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0109] The learning model generation device and the posture determination device can be realized not by a computer with a program installed, but by hardware corresponding to each part, for example, an electronic circuit. Furthermore, the learning model generation device and the posture determination device may be realized in part by a program and in the remaining part by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 12.

[0110] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0111] a parameter update unit that calculates, for each of the set combinations, a difference between the calculated similarity and the skeletal similarity, and updates parameters of the first machine learning model and the second machine learning model using the calculated difference. (Supplementary Note 1) A learning model generation device comprising: a training data acquisition unit that acquires training data including image data of a person and skeletal information of the person in the image data; an image feature extraction unit that extracts features of the image data from the image data using a first machine learning model; a skeletal feature extraction unit that extracts features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation unit that sets combinations of the image data and the skeletal information and calculates a similarity between the features of the image data and the features of the skeletal information for each of the set combinations; a corresponding skeletal information acquisition unit that acquires corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation unit that calculates, for each of the set combinations, a similarity between the corresponding skeletal information and the skeletal information as a skeletal similarity; and

[0112] (Supplementary Note 2) The learning model generation device according to Supplementary Note 1, wherein the first machine learning model learns the relationship between image data and its feature values, and the second machine learning model learns the relationship between skeletal information and its feature values.

[0113] (Supplementary Note 3) The learning model generation device according to Supplementary Note 1, wherein in the training data, the image data and the skeletal information corresponding to each other are linked together.

[0114] (Supplementary Note 4) The learning model generation device according to Supplementary Note 3, wherein the corresponding skeleton information acquisition unit acquires the skeleton information linked to the image data as the corresponding skeleton information.

[0115] (Supplementary Note 5) A posture determination device comprising: an image feature extraction unit that extracts image features of image data of a person who is the subject of posture determination, using a first machine learning model that has learned the relationship between image data and its features through machine learning; a skeletal feature extraction unit that extracts skeletal features of each piece of skeletal information for each of a plurality of predefined postures, using a second machine learning model that has learned the relationship between human skeletal information and its features through machine learning; a similarity calculation unit that calculates a similarity between the extracted image features and each of the extracted skeletal features; and a posture determination unit that identifies the skeletal feature with the highest similarity, further identifies a posture corresponding to the identified skeletal feature, and determines the identified posture as the posture of the person who is the subject of posture determination.

[0116] (Supplementary Note 6) The posture determination device according to Supplementary Note 5, wherein the parameters of the first machine learning model and the second machine learning model are updated by: calculating, for each combination of sample image data and sample skeletal information, a similarity between the features of the sample image data and the features of the sample skeletal information; further, for each combination, calculating a similarity between the skeletal information and corresponding skeletal information corresponding to the person in the sample image data as a skeletal similarity; and further, for each combination, calculating a difference between the calculated similarity and the skeletal similarity and using the calculated difference.

[0117] a parameter update step of calculating a difference between the calculated similarity and the skeletal similarity for each of the set combinations. (Supplementary Note 7) A learning model generation method comprising: a training data acquisition step of acquiring training data including image data of a person and skeletal information of the person in the image data; an image feature extraction step of extracting features of the image data from the image data using a first machine learning model; a skeletal feature extraction step of extracting features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation step of setting combinations of the image data and the skeletal information and calculating a similarity between a feature of the image data and a feature of the skeletal information for each of the set combinations; a corresponding skeletal information acquisition step of acquiring corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation step of calculating a similarity between the corresponding skeletal information and the skeletal information for each of the set combinations; and a parameter update step of calculating a difference between the calculated similarity and the skeletal similarity for each of the set combinations, and updating parameters of the first machine learning model and the second machine learning model using the calculated difference.

[0118] (Supplementary Note 8) The learning model generation method according to Supplementary Note 7, wherein the first machine learning model learns the relationship between image data and its features, and the second machine learning model learns the relationship between skeletal information and its features.

[0119] (Supplementary Note 9) The learning model generation method according to Supplementary Note 7, wherein in the training data, the image data and the skeletal information that correspond to each other are linked together.

[0120] (Supplementary Note 10) The learning model generation method according to Supplementary Note 9, wherein in the corresponding skeleton information acquisition step, the skeleton information linked to the image data is acquired as the corresponding skeleton information.

[0121] a parameter update step of calculating a difference between the calculated similarity and the skeletal similarity for each of the set combinations; and a parameter update step of updating parameters of the first machine learning model and the second machine learning model using the calculated similarity for each of the set combinations. (Supplementary Note 11) A computer-readable recording medium having recorded thereon a program including instructions to cause a computer to execute: a training data acquisition step of acquiring training data including image data of a person and skeletal information of the person in the image data; an image feature extraction step of extracting features of the image data from the image data using a first machine learning model; a skeletal feature extraction step of extracting features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation step of setting combinations of the image data and the skeletal information and calculating a similarity between a feature of the image data and a feature of the skeletal information for each of the set combinations; a corresponding skeletal information acquisition step of acquiring corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation step of calculating a similarity between the corresponding skeletal information and the skeletal information for each of the set combinations; and

[0122] (Supplementary Note 12) The computer-readable recording medium according to Supplementary Note 11, wherein the first machine learning model learns the relationship between image data and its feature values, and the second machine learning model learns the relationship between skeletal information and its feature values.

[0123] (Supplementary Note 13) The computer-readable recording medium according to Supplementary Note 11, wherein in the training data, the image data and the skeletal information corresponding to each other are linked together.

[0124] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 13, wherein in the corresponding skeleton information obtaining step, the skeleton information linked to the image data is obtained as the corresponding skeleton information.

[0125] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0126] This application claims priority based on Japanese Patent Application No. 2024-103046, filed on June 26, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0127] As described above, according to the present disclosure, in determining a person's posture using a machine learning model, it is possible to increase the degree of freedom in training data while improving the determination accuracy. The present disclosure is useful in a system for determining a person's posture from an image.

[0128] REFERENCE SIGNS LIST 10 Learning model generation device 11 Training data acquisition unit 12 Image feature extraction unit 13 Skeleton feature extraction unit 14 Similarity calculation unit 15 Corresponding skeleton information acquisition unit 16 Skeleton similarity calculation unit 17 Parameter update unit 21 First machine learning model 22 Second machine learning model 30 Database 40 Posture determination device 41 Image feature extraction unit 42 Skeleton feature extraction unit 43 Similarity calculation unit 44 Posture determination unit 45 Input reception unit 46 Memory unit 51 First machine learning model 52 Second machine learning model 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. A learning model generation device comprising: a training data acquisition means for acquiring training data including image data of a person and skeletal information of the person in the image data; an image feature extraction means for extracting features of the image data from the image data using a first machine learning model; a skeletal feature extraction means for extracting features of the skeletal information from the skeletal information using a second machine learning model; a similarity calculation means for setting combinations of the image data and the skeletal information and calculating a similarity between the features of the image data and the features of the skeletal information for each set combination; a corresponding skeletal information acquisition means for acquiring corresponding skeletal information corresponding to the person in the image data; a skeletal similarity calculation means for calculating a similarity between the corresponding skeletal information and the skeletal information for each set combination; and a parameter update means for calculating a difference between the calculated similarity and the skeletal similarity for each set combination, and updating parameters of the first machine learning model and the second machine learning model using the calculated difference.

2. The learning model generation device according to claim 1, wherein the first machine learning model learns the relationship between image data and its features, and the second machine learning model learns the relationship between skeletal information and its features.

3. The learning model generation device according to claim 1, wherein in the training data, the image data and the skeletal information that correspond to each other are linked together.

4. The learning model generation device according to claim 3, wherein the corresponding skeletal information acquisition means acquires the skeletal information linked to the image data as the corresponding skeletal information.

5. A posture determination device comprising: an image feature extraction means for extracting image features of image data of a person to be subjected to posture determination using a first machine learning model that has learned the relationship between image data and its features through machine learning; a skeletal feature extraction means for extracting skeletal features of each piece of skeletal information for each of a plurality of predefined postures using a second machine learning model that has learned the relationship between a person's skeletal information and its features through machine learning; a similarity calculation means for calculating the similarity between the extracted image features and each of the extracted skeletal features; and a posture determination means for identifying the skeletal feature with the highest similarity, further identifying the posture corresponding to the identified skeletal feature, and determining the identified posture as the posture of the person to be subjected to posture determination.

6. The posture determination device of claim 5, wherein the parameters of each of the first machine learning model and the second machine learning model are updated by: calculating, for each combination of sample image data and sample skeletal information, a similarity between the features of the sample image data and the features of the sample skeletal information; further, for each combination, calculating a skeletal similarity between the skeletal information and corresponding skeletal information corresponding to the person in the sample image data; and further, calculating, for each combination, a difference between the calculated similarity and the skeletal similarity, and using the calculated difference.

7. A learning model generation method comprising: acquiring training data including image data of a person and skeletal information of the person in the image data; extracting features of the image data from the image data using a first machine learning model; extracting features of the skeletal information from the skeletal information using a second machine learning model; setting combinations of the image data and the skeletal information; calculating, for each set combination, a similarity between the features of the image data and the features of the skeletal information; acquiring corresponding skeletal information corresponding to the person in the image data; calculating, for each set combination, the similarity between the corresponding skeletal information and the skeletal information as a skeletal similarity; calculating, for each set combination, a difference between the calculated similarity and the skeletal similarity; and using the calculated difference to update parameters of the first machine learning model and parameters of the second machine learning model.

8. The learning model generation method described in claim 7, wherein the first machine learning model learns the relationship between image data and its features, and the second machine learning model learns the relationship between skeletal information and its features.

9. The learning model generation method according to claim 7, wherein in the training data, the image data and the skeletal information that correspond to each other are linked together.

10. A learning model generation method according to claim 9, wherein, in acquiring the corresponding skeletal information, the skeletal information linked to the image data is acquired as the corresponding skeletal information.

11. A computer-readable recording medium having recorded thereon a program that causes a computer to: acquire training data including image data of a person and skeletal information of the person in the image data; extract features of the image data from the image data using a first machine learning model; extract features of the skeletal information from the skeletal information using a second machine learning model; set combinations of the image data and the skeletal information, and calculate, for each set combination, a similarity between the features of the image data and the features of the skeletal information; acquire corresponding skeletal information corresponding to the person in the image data; calculate, for each set combination, the similarity between the corresponding skeletal information and the skeletal information as skeletal similarity; calculate, for each set combination, a difference between the calculated similarity and the skeletal similarity; and use the calculated difference to update parameters of the first machine learning model and the second machine learning model.

12. The computer-readable recording medium according to claim 11, wherein the first machine learning model learns the relationship between image data and its features, and the second machine learning model learns the relationship between skeletal information and its features.

13. The computer-readable recording medium according to claim 11, wherein in the training data, the image data and the skeletal information that correspond to each other are linked together.

14. A computer-readable recording medium according to claim 13, wherein the program causes the computer to acquire, in acquiring the corresponding skeletal information, the skeletal information linked to the image data as the corresponding skeletal information.

Images