Processing device, processing method, and recording medium

Abstract

A processing device according to the present disclosure comprises an acquisition unit, a normalization unit, a reconstruction unit, an orientation adjustment unit, and an output unit. The acquisition unit acquires first and second models of interest. The normalization unit normalizes vectors which indicate respective states of a plurality of links in each of the first and second models of interest. The reconstruction unit generates first and second reconstructed models by reconstructing the first and second models of interest on the basis of the normalized vectors. The orientation adjustment unit adjusts the relative orientations of the first and second reconstructed models on the basis of the positional relationship between key points of the same type. The output unit outputs information pertaining to the first and second reconstructed models of which orientations have been adjusted.

Description

Processing device, processing method, and recording medium

[0001] The present disclosure relates to a processing device, a processing method, and a program.

[0002] A technology related to the present disclosure is disclosed in Patent Literature 1. The technology disclosed in Patent Literature 1 acquires skeletal data of an instructor and skeletal data of a user being trained by the instructor. The technology then normalizes at least one of the skeletal data so that the sizes of the two pieces of skeletal data correspond to each other. Normalization of skeletal data is a process of scaling one piece of skeletal data at a predetermined scale ratio to make the sizes of the two pieces of skeletal data correspond to each other. The technology then displays the normalized skeletal data of the instructor and the skeletal data of the user superimposed on each other.

[0003] Japanese Patent Application Laid-Open No. 2022-39120

[0004] The similarity between the postures of two objects may be determined using an object model that indicates the posture of the object using multiple key points and multiple links connecting two key points. There are various ways to determine the similarity. For example, as disclosed in Patent Document 1, two object models may be displayed superimposed on each other, and a person may visually compare the two object models to determine the similarity. Alternatively, a computer may calculate the similarity between two object models using a predetermined algorithm.

[0005] When determining the similarity between the postures of two objects using object models in this way, if the skeletons (lengths of links, ratios between links, etc.) or orientations of the two object models are different from each other, the accuracy of the determination may be poor. Patent Document 1 does not disclose this problem or a means for solving it.

[0006] In view of the above-mentioned problems, one example of the objective of the present disclosure is to provide a processing device, a processing method, and a program that improve the accuracy of determination in a process of determining the similarity between the poses of two objects using an object model.

[0007] According to the present disclosure, there is provided a processing device having: an acquisition means for acquiring a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; a normalization means for normalizing a vector that indicates the state of each of the plurality of links in each of the first object model and the second object model; a reconstruction means for generating a first reconstructed model that reconstructs the first object model based on the vectors after normalization, and a second reconstructed model that reconstructs the second object model based on the vectors after normalization; an orientation adjustment means for adjusting the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and an output means for outputting information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

[0008] The present disclosure also provides a processing method in which one or more computers acquire a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points, normalize vectors that indicate the state of each of the plurality of links in each of the first object model and the second object model, generate a first reconstructed model that reconstructs the first object model based on the normalized vectors, and generate a second reconstructed model that reconstructs the second object model based on the normalized vectors, adjust the relative orientations of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type, and output information related to the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

[0009] Furthermore, according to the present disclosure, there is provided a program that causes a computer to function as: an acquisition means that acquires a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; a normalization means that normalizes a vector that indicates the state of each of the plurality of links in each of the first object model and the second object model; a reconstruction means that generates a first reconstructed model that reconstructs the first object model based on the vectors after normalization, and a second reconstructed model that reconstructs the second object model based on the vectors after normalization; an orientation adjustment means that adjusts the relative orientations of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and an output means that outputs information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

[0010] According to one aspect of the present disclosure, a processing device, a processing method, and a program are provided that improve the accuracy of determination in a process of determining the similarity between the poses of two objects using an object model.

[0011] FIG. 1 is a diagram illustrating an example of a functional block diagram of a processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an example of a processing flow of a processing device according to the present disclosure. FIG. 3 is a diagram illustrating an overview of processing of a processing device according to the present disclosure. FIG. 4 is a diagram illustrating an example of a hardware configuration of a processing device according to the present disclosure. FIG. 5 is a diagram illustrating an example of a target model. FIG. 6 is a diagram illustrating an example of a vector direction. FIG. 7 is a diagram illustrating an example of information processed by a processing device according to the present disclosure. FIG. 8 is a diagram illustrating an example of a problem solved by a processing device according to the present disclosure. FIG. 9 is another diagram illustrating an example of a problem solved by a processing device according to the present disclosure. FIG. 10 is another diagram illustrating another example of a problem solved by a processing device according to the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0013] First Embodiment First, the above-mentioned problem will be described in more detail. Specifically, the following will be described with reference to drawings: "When determining the similarity between the postures of two objects using object models, if the skeletons (lengths of links, ratios between links, etc.) or orientations of the two object models are different from each other, the accuracy of the determination may be reduced."

[0014] Three object models are shown in Figure 8. The object models indicate the posture of the object using multiple key points (black circles in the figure) and multiple links (lines connecting two black circles) connecting two key points. The key points correspond to, for example, but are not limited to, the head, neck, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right hip, left hip, abdomen, right knee, left knee, right foot, and left foot.

[0015] Here, target models A to C all have the same posture. In other words, when comparing the angles between two adjacent links, the angles formed by the same type of links in each of target models A to C match. However, the "skeleton" of target models A to C is different from one another. In other words, the lengths of the same type of links do not match completely. Furthermore, the ratio of the length of one type of link to the length of another type of link does not match completely. For example, target models A and B have different lengths of the links connecting the key points of the neck and abdomen (corresponding to the length of the torso).

[0016] Thus, even if the poses are the same, if the skeletons of the target models are different, the accuracy of similarity determination by a human visually or similarity calculation by a computer using a predetermined algorithm may be poor. For example, even if target models A to C are displayed superimposed on each other, key points of the same type do not completely overlap. Figure 9 shows the superimposed state of target models A and B. The link of target model B is indicated by a dotted line. As shown in the figure, when key points of the same type do not completely overlap, it is difficult to determine whether target models A and B have the same pose.

[0017] Next, two other target models are shown in Figure 10. Target models D and E have the same posture. However, the orientations of target models D and E are different. Specifically, target model E is rotated counterclockwise by a predetermined angle from target model D.

[0018] In this way, even if the poses are the same, if the orientations of the target models are different, the accuracy of similarity determination by a human visually or similarity calculation by a computer using a predetermined algorithm may be poor. For example, even if target models D and E are displayed superimposed on each other, key points of the same type do not completely overlap. Figure 11 shows the superimposed target models D and E. The link of target model E is indicated by a dotted line. As shown in the figure, when key points of the same type do not completely overlap, it is difficult to determine whether target models D and E have the same pose.

[0019] Next, a processing apparatus 10 according to the present embodiment, which is configured to be able to solve such problems, will be described.

[0020] Fig. 1 is a functional block diagram showing an overview of a processing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the processing device 10.

[0021] 1, the processing device 10 includes an acquisition unit 11, a normalization unit 12, a reconstruction unit 13, a direction adjustment unit 14, and an output unit 15. These functional units execute the processing of the flowchart in FIG.

[0022] In S10, the acquisition unit 11 acquires a first object model and a second object model that indicate the posture of the object using a plurality of key points and a plurality of links connecting two key points.

[0023] In S11, the normalization unit 12 normalizes vectors indicating the states of each of the multiple links in each of the first and second object models.

[0024] In S12, the reconstruction unit 13 generates a first reconstruction model by reconstructing the first target model based on the normalized vector, and a second reconstruction model by reconstructing the second target model based on the normalized vector.

[0025] In S13, the orientation adjustment unit 14 adjusts the relative orientations of the first reconstructed model and the second reconstructed model based on the positional relationship between key points of the same type.

[0026] In S14, the output unit 15 outputs information relating to the first reconstructed model after the orientation adjustment and the second reconstructed model after the orientation adjustment.

[0027] In this way, the processing device 10 normalizes each of the first and second target models by individually normalizing the vectors indicating the state of each of the multiple links in the target model (by making the link lengths uniform). With this processing device 10, the lengths of links of the same type can be made uniform in the first and second target models, thereby making the length ratios between multiple types of links uniform. In this way, the processing device 10 aligns the skeletons of the first and second target models by normalizing the link lengths through the vector normalization. Note that, in the case of normalization in which skeleton data is scaled at a predetermined scale, as in the technology disclosed in Patent Document 1, the length ratios between multiple types of links constituting the target model remain unchanged before and after normalization. In the normalization disclosed in Patent Document 1, the size of the skeleton data is changed, but the skeleton (the ratio between links) is not changed.

[0028] Furthermore, the processing device 10 can adjust the relative orientation of the first reconstructed model and the second reconstructed model, which have the same link length, thereby eliminating any misalignment in the relative orientation between the first reconstructed model and the second reconstructed model.

[0029] Furthermore, the processing device 10 can perform a process of adjusting the relative orientations of the first reconstructed model and the second reconstructed model after performing a normalization process to make the lengths of the links uniform. By adjusting the orientations after normalization, the orientations can be adjusted with high precision.

[0030] The processing device 10 can eliminate differences in skeleton and orientation that may exist between two object models, thereby improving the accuracy of the process of determining the similarity between the postures of two objects using the object models.

[0031] <<Second Embodiment>> <Overview> The processing device 10 of the second embodiment is a specific implementation of the configuration of the processing device 10 of the first embodiment. An overview of the processing performed by the processing device 10 will be described with reference to FIG. 3 .

[0032] First, the processing device 10 acquires a first object model and a second object model to be compared with each other. Then, the processing device 10 performs a process of aligning the lengths of like links in the acquired first object model and second object model. Through this process, a first reconstructed model is generated from the first object model, and a second reconstructed model is generated from the second object model. The first reconstructed model and the second reconstructed model, which have the same lengths of like links, have the same skeleton.

[0033] Next, the processing device 10 performs a process of adjusting the relative orientations of the first reconstructed model and the second reconstructed model, so that the orientations of the first reconstructed model and the second reconstructed model are aligned.

[0034] These processes eliminate the differences in skeleton (link length, link ratio, etc.) and orientation that existed between the first and second target models. By referring to the "first reconstructed model and second reconstructed model after orientation adjustment" in FIG. 3 , in which the differences in skeleton and orientation have been eliminated, the similarity of the postures of the first and second target models can be easily determined. In the example of FIG. 3 , the positions of the key points corresponding to the "right foot" are shifted from each other. The positions of the key points corresponding to other parts are identical (overlapping with each other). From this, it is visually and easily apparent that the postures of the first and second target models differ in the right foot area and are identical in other parts.

[0035] <Hardware Configuration> Next, an example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-stored in the device before shipping, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0036] FIG. 4 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 4, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0037] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0038] <Functional Configuration> Next, a detailed description will be given of the functional configuration of the processing device 10. Fig. 1 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, a normalization unit 12, a reconstruction unit 13, an orientation adjustment unit 14, and an output unit 15.

[0039] The acquisition unit 11 acquires a first target model and a second target model. The first target model and the second target model may be two-dimensional models or three-dimensional models.

[0040] An "object model" indicates the posture of an object using multiple key points and multiple links connecting two key points. The object model is expressed, for example, by information indicating the relative positional relationship between multiple key points, or information indicating pairs of key points connected to each other by links. An "object" is, but is not limited to, a person, other animals, plants, objects, etc. In this embodiment, the object is assumed to be a person.

[0041] An example of an object model is shown in Figure 5. Figure 5 is an object model showing the posture of a person. Key points are indicated by black circles, and links are indicated by straight lines. The object model in Figure 5 includes 15 key points corresponding to the head, neck, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right hip, left hip, abdomen, right knee, left knee, right foot, and left foot. The number of key points and which parts the key points correspond to are not limited to this example.

[0042] The acquisition unit 11 can acquire the object model by various means. For example, the acquisition unit 11 may acquire an image input by a user. The acquisition unit 11 may then analyze the image to estimate the posture of an object included in the image and generate an object model indicating the posture of the object. The acquisition unit 11 can estimate the posture of the object by detecting key points of the object included in the image using well-known technology such as Openpose.

[0043] Alternatively, the acquisition unit 11 may output a UI (user interface) screen showing the target model as shown in FIG. 5 . The UI screen can be output via an output device such as a display or a projection device. Furthermore, if the processing device 10 is a server, the UI screen may be output by transmitting the UI screen to a client terminal and displaying it. The acquisition unit 11 then accepts a user operation, such as moving the positions of key points, for the target model as shown in FIG. 5 displayed on the UI screen. In response to this user operation, the positions of the key points displayed on the UI screen move, and the posture of the target model displayed on the UI screen changes accordingly. In this way, the user adjusts the posture of the target model displayed on the UI screen by moving the positions of the key points displayed on the UI screen. The acquisition unit 11 can then acquire the target model in a predetermined posture determined by the user operation. The user operation is realized via an input device such as a touch panel, a keyboard, a mouse, a physical button, or a microphone. Furthermore, if the processing device 10 is a server, the user operation may be realized via a client terminal.

[0044] Alternatively, the acquisition unit 11 may acquire the target model stored in an external device by any means at any timing.

[0045] "Acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request to another device and receiving a response, and accessing and reading information from another device or storage medium. An example of passive acquisition is receiving information that is distributed (or transmitted, push notification, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0046] 1 , the normalization unit 12 normalizes vectors indicating the states of each of the multiple links in each of the first and second object models. The normalization unit 12 can normalize the multiple vectors individually. The state of a link is represented by the length and direction of the link.

[0047] First, the process of calculating the vector will be described.

[0048] The normalization unit 12 calculates one vector for each link. In the case of a target model having 14 links as in the example of Fig. 5, the normalization unit 12 calculates 14 vectors. Each vector indicates the length and direction of the link.

[0049] The direction of a link is the direction from the key point at one end of the link to the key point at the other end. There are various ways to determine which key point the direction of each link represented by a vector should be based on, but an example will be explained below.

[0050] In one example, one of the multiple key points is predetermined as the reference key point. In this embodiment, the abdominal key point is set as the reference key point S as shown in Fig. 6, but this is not limiting. Then, in this example, the normalization unit 12 calculates a vector having a directional component pointing in a direction away from the reference key point S, as indicated by the arrow in Fig. 6.

[0051] Next, the process of normalizing a vector will be described.

[0052] The normalization unit 12 normalizes each vector by modifying the length component while maintaining (not changing) the directional component. For example, the normalization unit 12 normalizes multiple vectors by converting the values ​​of the length components to the same predetermined value while maintaining the directional components. As a result of this normalization, the length components of multiple vectors calculated from one target model have the same value. Furthermore, the length components of vectors calculated from each of multiple target models also have the same value.

[0053] The reconstruction unit 13 generates a first reconstructed model by reconstructing the first object model based on the normalized vector. If the first object model is a two-dimensional model, the first reconstructed model is a two-dimensional model. If the first object model is a three-dimensional model, the first reconstructed model is a three-dimensional model.

[0054] The reconstruction unit 13 generates a first reconstructed model by normalizing each of a plurality of vectors calculated from the first target model and connecting the normalized vectors. The reconstruction unit 13 connects the normalized vectors in the same order as the order of the plurality of vectors in the first target model. There are various ways to connect the vectors. For example, the reconstruction unit 13 may generate the first reconstructed model by connecting the normalized vectors in order from a reference keypoint in a direction away from the reference point.

[0055] The reconstruction unit 13 also generates a second reconstructed model by reconstructing the second object model based on the normalized vector. If the second object model is a two-dimensional model, the second reconstructed model is a two-dimensional model. If the second object model is a three-dimensional model, the second reconstructed model is a three-dimensional model.

[0056] The reconstruction unit 13 generates a second reconstructed model by normalizing each of the plurality of vectors calculated from the second object model and connecting the normalized vectors. The reconstruction unit 13 connects the normalized vectors in the same order as the order of the plurality of vectors in the second object model. There are various ways to connect the vectors. For example, the connecting method described in the above-mentioned method for generating the first reconstructed model may be adopted.

[0057] The processing by the normalization unit 12 and the reconstruction unit 13 described above can be said to be processing for generating a first reconstruction model by aligning the lengths of the multiple links constituting the first target model to a predetermined length while maintaining the identity of the posture of the first target model.

[0058] Similarly, the processing by the normalization unit 12 and the reconstruction unit 13 described above can be said to be processing for generating a second reconstructed model by aligning the lengths of the multiple links constituting the second target model to a predetermined length while maintaining the identity of the posture of the second target model.

[0059] "Maintaining the same posture" means maintaining the orientation of the link, maintaining the angle between two adjacent links, etc. "Aligning the lengths of the links to a predetermined length" means making the lengths of the links the same value, or keeping the difference in link length within a standard range, etc.

[0060] Returning to FIG. 1, the orientation adjustment unit 14 adjusts the relative orientations of the first reconstructed model and the second reconstructed model based on the positional relationship between key points of the same type.

[0061] "Keypoints of the same type" are keypoints that correspond to the same part (same location) of the object. For example, the object model in FIG. 5 has keypoints that correspond to the head, neck, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right hip, left hip, abdomen, right knee, left knee, right foot, and left foot. For example, the keypoints that correspond to the head of the first reconstructed model and the keypoints that correspond to the head of the second reconstructed model are the same type of keypoints. On the other hand, the keypoints that correspond to the head of the first reconstructed model and the keypoints that correspond to the neck of the second reconstructed model are different types of keypoints.

[0062] "Links of the same type" are links corresponding to the same part (same location) of the object. For example, the object model in FIG. 5 has key points corresponding to the head, neck, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right hip, left hip, abdomen, right knee, left knee, right foot, and left foot, respectively, and a plurality of links connecting two key points. For example, a link connecting the head key point and neck key point of the first reconstructed model and a link connecting the head key point and neck key point of the second reconstructed model are the same type of link. On the other hand, a link connecting the head key point and neck key point of the first reconstructed model and a link connecting the neck key point and right shoulder key point of the second reconstructed model are different types of links.

[0063] The "positional relationship between keypoints of the same type" refers to at least one of the distance between keypoints of the same type and whether or not keypoints of the same type overlap each other.

[0064] When calculating the positional relationship, the orientation adjustment unit 14 places the first reconstructed model and the second reconstructed model in a state where they can be compared with each other. The state where they can be compared with each other means, for example, positioning the first reconstructed model and the second reconstructed model in the same coordinate system (a two-dimensional coordinate system or a three-dimensional coordinate system). In this state, the orientation adjustment unit 14 can manage the coordinates of various key points of the first reconstructed model. The orientation adjustment unit 14 can also manage the coordinates of various key points of the second reconstructed model.

[0065] After the first and second reconstructed models are made comparable to each other, the orientation adjustment unit 14 calculates the positional relationship between the same type of keypoints as described above. The orientation adjustment unit 14 can calculate the positional relationship between the same type of keypoints as described above, for example, based on the coordinates of the various keypoints in the first reconstructed model and the coordinates of the various keypoints in the second reconstructed model.

[0066] The "adjustment of the relative orientation of the first reconstructed model and the second reconstructed model" is realized by rotating at least one of the first reconstructed model and the second reconstructed model around a reference keypoint. That is, the orientation adjustment unit 14 performs such a rotation process on at least one of the first reconstructed model and the second reconstructed model.

[0067] Alternatively, the orientation adjustment unit 14 may translate at least one of the first reconstructed model and the second reconstructed model. For example, the orientation adjustment unit 14 may translate at least one of the first reconstructed model and the second reconstructed model to overlap their reference keypoints. That is, the orientation adjustment unit 14 may translate at least one of the first reconstructed model and the second reconstructed model so that the reference keypoints of the first reconstructed model and the reference keypoints of the second reconstructed model have the same coordinates. The orientation adjustment unit 14 may perform the rotation process after performing the translation.

[0068] The orientation adjustment unit 14 can adjust the relative orientations of the first reconstructed model and the second reconstructed model so that the relative orientations of the first reconstructed model and the second reconstructed model satisfy a predetermined orientation adjustment condition. For example, the orientation adjustment unit 14 may translate at least one of the first reconstructed model and the second reconstructed model to align their reference keypoints, and then adjust the relative orientations of the first reconstructed model and the second reconstructed model by the rotation process.

[0069] The orientation adjustment condition is one of the following: ・The total distance between keypoints of the same type is the smallest ・The total distance between keypoints of a certain set of types is the smallest ・The distance between keypoints of a certain number of types is below a threshold ・The number of types of keypoints whose distance from each other is below a threshold is the largest ・The number of types of keypoints that overlap each other is the largest ・A condition combining two or more of the above five conditions with a logical operator

[0070] "The sum of distances between keypoints of the same type is the smallest" In this example, the orientation adjustment unit 14 calculates the distance between the position (coordinates) in the first reconstructed model and the position (coordinates) in the second reconstructed model for each type of keypoint. Then, the orientation adjustment unit 14 obtains the sum of the distances calculated for each type of keypoint as a reference value.

[0071] The orientation adjustment unit 14 rotates at least one of the first reconstructed model and the second reconstructed model around the reference keypoint to find a state where the reference value is minimized. This state is a state that satisfies the orientation adjustment condition.

[0072] "The sum of distances between a predetermined set of key points is minimum" In this example, a set of key points from among multiple types of key points are designated in advance as reference targets. Then, the orientation adjustment unit 14 calculates the distance between the position (coordinates) in the first reconstructed model and the position (coordinates) in the second reconstructed model for each of the key points of the reference target. Then, the orientation adjustment unit 14 obtains the sum of the distances calculated for each of the key points of the reference target as the reference value.

[0073] The orientation adjustment unit 14 rotates at least one of the first reconstructed model and the second reconstructed model around the reference keypoint to find a state where the reference value is minimized. This state is a state that satisfies the orientation adjustment condition.

[0074] "The distance between a predetermined number or more types of keypoints is equal to or less than the threshold value" In this example, the orientation adjustment unit 14 calculates the distance between the position (coordinates) in the first reconstructed model and the position (coordinates) in the second reconstructed model for each type of keypoint. Then, the orientation adjustment unit 14 compares the distance with the threshold value for each type of keypoint, and based on the comparison result, counts the number of types of keypoints for which the distance is equal to or less than the threshold value.

[0075] The orientation adjustment unit 14 rotates at least one of the first reconstructed model and the second reconstructed model around the reference keypoint, and determines a state where the count value is equal to or greater than a predetermined number. This state satisfies the orientation adjustment condition.

[0076] "The largest number of types of keypoints whose mutual distance is equal to or less than the threshold" In this example, the orientation adjustment unit 14 calculates the distance between the position (coordinates) in the first reconstructed model and the position (coordinates) in the second reconstructed model for each type of keypoint. Then, the orientation adjustment unit 14 compares the distance with the threshold for each type of keypoint, and based on the comparison result, counts the number of types of keypoints whose distance is equal to or less than the threshold.

[0077] The orientation adjustment unit 14 rotates at least one of the first reconstructed model and the second reconstructed model around the reference keypoint to find the state where the count value is maximized. This state is the state that satisfies the orientation adjustment condition.

[0078] "The largest number of types of overlapping key points" In this example, the orientation adjustment unit 14 determines, for each type of key point, whether the position (coordinates) in the first reconstructed model and the position (coordinates) in the second reconstructed model match. Then, the orientation adjustment unit 14 counts the number of types of key points that are determined to match.

[0079] The orientation adjustment unit 14 rotates at least one of the first reconstructed model and the second reconstructed model around the reference keypoint to find the state where the count value is maximized. This state is the state that satisfies the orientation adjustment condition.

[0080] The output unit 15 outputs information relating to the first reconstructed model after the orientation adjustment and the second reconstructed model after the orientation adjustment.

[0081] The output unit 15 can output at least one of the following information: An image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are simultaneously displayed An image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are superimposed and displayed Coordinate information of each of a plurality of key points in each of the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment Similarity between the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment

[0082] The "image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are simultaneously displayed" may be an image in which the two reconstructed models are displayed in an overlapping manner, as shown as "first reconstructed model and second reconstructed model after orientation adjustment" in Fig. 3. Alternatively, although not shown, the image may be an image in which the two reconstructed models are displayed side by side without being overlapped.

[0083] The "coordinate information" indicates the coordinates of each of the multiple keypoints in the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment. As described above, the first reconstructed model and the second reconstructed model are placed in the same coordinate system and orientation adjustment is performed. The coordinate information indicates, for example, the coordinates in this coordinate system of each of the multiple keypoints in the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

[0084] The "similarity" is the similarity calculated based on the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment. There are various ways to calculate the similarity, and any widely known method can be used. For example, the similarity can be calculated based on the number of types of overlapping keypoints, the distance between keypoints of the same type, the total value of the distances between various keypoints, the difference in angle between two links of the same type, etc., but is not limited to these.

[0085] The output unit 15 may output the image and the similarity at the same time. That is, the output unit 15 may output a screen on which the image and the similarity are displayed at the same time.

[0086] The output unit 15 can output the image, the similarity, or a screen displaying the image and the similarity simultaneously via an output device such as a display or a projection device. Furthermore, if the processing device 10 is a server, the output unit 15 may transmit the image, the similarity, or a screen displaying the image and the similarity simultaneously to a client terminal and cause the client terminal to display the image. Furthermore, the output unit 15 may output a file indicating the coordinate information. The file format is not particularly limited. The user can store the file in a storage device within the processing device 10 or in a storage device of an external device.

[0087] Next, an example of the processing flow of the processing device 10 will be described using the flowchart of Fig. 2. Details of each process have been described above, so a description thereof will be omitted here.

[0088] In S10, the processing device 10 acquires a first object model and a second object model that indicate the posture of the object using a plurality of key points and a plurality of links connecting two key points.

[0089] In S11, the processing device 10 normalizes vectors indicating the states of each of the multiple links in each of the first object model and the second object model.

[0090] In S12, the processing device 10 generates a first reconstructed model by reconstructing the first object model based on the normalized vector, and a second reconstructed model by reconstructing the second object model based on the normalized vector.

[0091] In S13, the processing device 10 adjusts the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between keypoints of the same type.

[0092] In S14, the processing device 10 outputs information about the first reconstructed model after the orientation adjustment and the second reconstructed model after the orientation adjustment.

[0093] Effects and Advantages According to the processing apparatus 10 of this embodiment, the same effects and advantages as those of the first embodiment are achieved.

[0094] Furthermore, the processing device 10 can simultaneously display two object models after eliminating differences in the skeletons (link lengths, link ratios, etc.) and orientations between the two object models. For example, the processing device 10 can display the two reconstructed models in an overlapping manner, as shown as "first reconstructed model and second reconstructed model after orientation adjustment" in FIG. 3. With this processing device 10, the user can easily determine the similarity in the postures of the two object models based on the output information.

[0095] Furthermore, the processing device 10 can resolve differences in the skeletons (link lengths, link ratios, etc.) and orientations between the two target models, and then output coordinate information for multiple key points in each of the two target models. A user can apply the coordinate information to a predetermined program to execute desired processing. For example, the user can use the coordinate information to have a computer determine the similarity between the two target models.

[0096] Furthermore, the processing device 10 can align the skeletons of the two target models by generating a reconstructed model with the same link lengths using the characteristic processing described above. The processing device 10 employing the characteristic processing described above can align the skeletons of the two target models with high accuracy.

[0097] Furthermore, the processing device 10 can align the orientations of the two target models by the characteristic processing described above. According to the processing device 10 employing the characteristic processing described above, the orientations of the two target models can be align with high accuracy.

[0098] <Modifications> Here, modifications applicable to the processing apparatus 10 of the first and second embodiments will be described.

[0099] In the second embodiment, the normalization unit 12 normalizes the length components of vectors of all of the multiple types of links included in one target model to the same value. This normalization is equivalent to making the lengths of all of the multiple types of links included in one target model the same length.

[0100] In the modified example, the normalization unit 12 normalizes the length components of the vectors of the multiple types of links included in one target model to predetermined values ​​corresponding to the various types of links. That is, the normalization unit 12 converts the length components of the vectors of the multiple types of links included in one target model to predetermined values ​​according to the type of link while maintaining the orientation components.

[0101] There are various specific steps for this normalization.

[0102] In one example, the normalization unit 12 first sets the length components of vectors of all multiple types of links included in one target model to the same value using the method described in the second embodiment. Then, the normalization unit 12 normalizes the vectors of each link by multiplying the length component values ​​of the vectors by predetermined coefficients corresponding to each link. In this example, as shown in FIG. 7 , coefficients associated with each link are determined in advance and stored in a predetermined storage device. The storage device may be provided within the processing device 10 or in an external device accessible from the processing device 10. The normalization unit 12 performs the above normalization using information such as that shown in FIG. 7 .

[0103] In another example, predetermined length component values ​​are determined in advance in association with various links and stored in a predetermined storage device. The storage device may be provided within the processing device 10 or in an external device accessible from the processing device 10. The normalization unit 12 performs the normalization by converting the length component of the vector of each of the multiple types of links calculated from the target model into values ​​corresponding to the various links indicated by the information.

[0104] According to this modification, the skeleton of the object shown in the reconstructed model becomes more natural. That is, the balance between the lengths of the legs and arms, and the length of the torso and legs, etc. become more natural. As a result, the user can more easily grasp the content of the "image simultaneously displaying the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment" output by the output unit 15.

[0105] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0106] In addition, in the flowcharts used in the above explanation, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0107] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes: 1. A processing device comprising: an acquisition means for acquiring a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; a normalization means for normalizing a vector that indicates the state of each of the plurality of links in each of the first object model and the second object model; a reconstruction means for generating a first reconstructed model that reconstructs the first object model based on the vectors after normalization, and a second reconstructed model that reconstructs the second object model based on the vectors after normalization; an orientation adjustment means for adjusting the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and an output means for outputting information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment. 2. The processing device according to 1, wherein the output means outputs as the information at least one of: an image simultaneously displaying the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; an image superimposing the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; coordinate information of each of the plurality of keypoints in the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; and a similarity between the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment. 3. The processing device according to 1 or 2, wherein the normalization means, as the vector normalization, performs a process of converting a length component of each of the plurality of vectors to a predetermined value while maintaining the orientation component, or a process of converting a length component of each of the plurality of vectors to a predetermined value according to the type of the link while maintaining the orientation component. 4. The processing device according to any of 1 to 3, wherein the reconstruction means generates the first reconstructed model and the second reconstructed model by connecting the normalized vectors in order starting from a reference keypoint that is one of the plurality of keypoints.5. The processing device according to any one of 1 to 4, wherein the orientation adjustment means adjusts the relative orientation of the first reconstructed model and the second reconstructed model by rotating at least one of the first reconstructed model and the second reconstructed model around a reference keypoint that is one of the multiple keypoints. 6. The processing device according to any one of 1 to 5, wherein the orientation adjustment means adjusts the relative orientation of the first reconstructed model and the second reconstructed model so that the following conditions are satisfied: the total distance between the keypoints of the same type is minimum, the total distance between a predetermined portion of the keypoints is minimum, the distance between a predetermined number or more of the keypoints is equal to or less than a threshold, the number of types of the keypoints whose distances from each other are equal to or less than a threshold is maximum, or the number of types of the keypoints that overlap each other is maximum. 7. The processing device according to 5, wherein the orientation adjustment means translates at least one of the first reconstructed model and the second reconstructed model to overlap each other's reference keypoints, and then adjusts the relative orientation of the first reconstructed model and the second reconstructed model by the rotation. 8. 8. The processing device according to any one of 1 to 7, wherein the normalization means calculates, as the vector of each of the plurality of links, a vector having a directional component pointing in a direction away from a reference keypoint that is one of the plurality of keypoints. 9. The processing device according to any one of 1 to 8, wherein the first object model, the second object model, the first reconstructed model, and the second reconstructed model are two-dimensional models or three-dimensional models.10. A processing method in which one or more computers: acquire a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; normalize vectors that indicate the state of each of the plurality of links in each of the first object model and the second object model; generate a first reconstructed model that reconstructs the first object model based on the normalized vectors; and generate a second reconstructed model that reconstructs the second object model based on the normalized vectors; adjust the relative orientations of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and output information about the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment. a normalization means for normalizing vectors indicating the state of each of the plurality of links in each of the first and second object models; a reconstruction means for generating a first reconstructed model by reconstructing the first object model based on the vectors after normalization, and a second reconstructed model by reconstructing the second object model based on the vectors after normalization; an orientation adjustment means for adjusting the relative orientations of the first and second reconstructed models based on the positional relationship between the key points of the same type; and an output means for outputting information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

[0108] Some or all of Supplements 2 to 9 that are dependent on the processing device of Supplement 1 described above may also be dependent on the processing method of Supplement 10 and the program of Supplement 11 in the same dependent relationship as Supplement 1 and Supplements 2 to 9. Furthermore, within the scope of each of the above-described embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0109] This application claims priority based on Japanese Patent Application No. 2023-214485, filed December 20, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0110] 10 Processing device 11 Acquisition unit 12 Normalization unit 13 Reconstruction unit 14 Orientation adjustment unit 15 Output unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. A processing device having: an acquisition means for acquiring a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; a normalization means for normalizing a vector that indicates the state of each of the plurality of links in each of the first object model and the second object model; a reconstruction means for generating a first reconstructed model that reconstructs the first object model based on the vector after normalization, and a second reconstructed model that reconstructs the second object model based on the vector after normalization; an orientation adjustment means for adjusting the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and an output means for outputting information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

2. The processing device according to claim 1, wherein the output means outputs as the information at least one of the following: an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are simultaneously displayed; an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are displayed in an overlaid manner; coordinate information of each of the multiple key points in the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; and a similarity between the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

3. The processing device according to claim 1 or 2, wherein the normalization means performs the following to normalize the vectors: converting the length component of each of the plurality of vectors to a predetermined value while maintaining the orientation component; or converting the length component of each of the plurality of vectors to a predetermined value according to the type of link while maintaining the orientation component.

4. A processing device according to any one of claims 1 to 3, wherein the reconstruction means generates the first reconstruction model and the second reconstruction model by connecting the normalized vectors in order from a reference keypoint which is one of the multiple keypoints.

5. A processing device according to any one of claims 1 to 4, wherein the orientation adjustment means adjusts the relative orientation of the first reconstructed model and the second reconstructed model by rotating at least one of the first reconstructed model and the second reconstructed model around a reference keypoint, which is one of the multiple keypoints.

6. A processing device according to any one of claims 1 to 5, wherein the orientation adjustment means adjusts the relative orientations of the first reconstructed model and the second reconstructed model so as to satisfy the following conditions: the sum of the distances between the keypoints of the same type is a minimum; the sum of the distances between the keypoints of a predetermined portion of types is a minimum; the distances between the keypoints of a predetermined number or more types are equal to or less than a threshold; the number of types of the keypoints whose distances to each other are equal to or less than a threshold is the largest; or the number of types of the keypoints that overlap each other is the largest.

7. The processing device according to claim 5, wherein said orientation adjustment means translates at least one of the first reconstructed model and the second reconstructed model to align their reference key points, and then adjusts the relative orientations of the first reconstructed model and the second reconstructed model by the rotation.

8. A processing device according to any one of claims 1 to 7, wherein the normalization means calculates, as the vector for each of the plurality of links, a vector having a directional component pointing in a direction away from a reference key point which is one of the plurality of key points.

9. A processing device according to any one of claims 1 to 8, wherein the first object model, the second object model, the first reconstructed model, and the second reconstructed model are two-dimensional models or three-dimensional models.

10. A processing method in which one or more computers obtain a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; normalize vectors that indicate the state of each of the plurality of links in each of the first object model and the second object model; generate a first reconstructed model that reconstructs the first object model based on the normalized vectors, and a second reconstructed model that reconstructs the second object model based on the normalized vectors; adjust the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and output information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

11. The processing method according to claim 10, wherein the one or more computers, in outputting the information, output at least one of the following: an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are simultaneously displayed; an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are displayed in a superimposed manner; coordinate information of each of the multiple key points in each of the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; and a similarity between the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

12. The processing method according to claim 10 or 11, wherein the one or more computers execute, as the normalization of the vectors, a process of converting the length component of each of the plurality of vectors to a predetermined value while maintaining the orientation component, or a process of converting the length component of each of the plurality of vectors to a predetermined value according to the type of the link while maintaining the orientation component.

13. The processing method according to any one of claims 10 to 12, wherein the one or more computers generate the first reconstruction model and the second reconstruction model by connecting the normalized vectors in order from a reference keypoint that is one of the multiple keypoints.

14. A processing method according to any one of claims 10 to 13, wherein the one or more computers adjust the relative orientation of the first reconstructed model and the second reconstructed model by rotating at least one of the first reconstructed model and the second reconstructed model around a reference keypoint that is one of the multiple keypoints.

15. A processing method according to any one of claims 10 to 14, wherein the one or more computers adjust the relative orientations of the first reconstructed model and the second reconstructed model so as to satisfy the following: a sum of distances between the keypoints of the same type is a minimum; a sum of distances between the keypoints of a predetermined portion of types is a minimum; a distance between the keypoints of a predetermined number or more types is less than or equal to a threshold; the number of types of the keypoints whose distances to each other are less than or equal to a threshold is the largest; or the number of types of the keypoints that overlap each other is the largest.

16. A recording medium having recorded thereon a program that causes a computer to function as: an acquisition means for acquiring a first object model and a second object model that indicate the posture of an object using a plurality of key points and a plurality of links connecting two of the key points; a normalization means for normalizing a vector that indicates the state of each of the plurality of links in each of the first object model and the second object model; a reconstruction means for generating a first reconstructed model that reconstructs the first object model based on the vector after normalization, and a second reconstructed model that reconstructs the second object model based on the vector after normalization; an orientation adjustment means for adjusting the relative orientation of the first reconstructed model and the second reconstructed model based on the positional relationship between the key points of the same type; and an output means for outputting information regarding the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

17. The recording medium according to claim 16, wherein the output means records the program for outputting as the information at least one of the following: an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are simultaneously displayed; an image in which the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment are superimposed; coordinate information of each of the multiple key points in each of the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment; and a similarity between the first reconstructed model after orientation adjustment and the second reconstructed model after orientation adjustment.

18. The recording medium according to claim 16 or 17, wherein the normalization means records the program for executing the following process to normalize the vector: converting the length component of each of the plurality of vectors to a predetermined value while maintaining the orientation component; or converting the length component of each of the plurality of vectors to a predetermined value according to the type of link while maintaining the orientation component.

19. A recording medium according to any one of claims 16 to 18, wherein the reconstruction means records the program for generating the first reconstruction model and the second reconstruction model by connecting the normalized vectors in order from a reference keypoint which is one of the multiple keypoints.

20. A recording medium according to any one of claims 16 to 19, wherein the orientation adjustment means records the program for adjusting the relative orientation of the first reconstructed model and the second reconstructed model by rotating at least one of the first reconstructed model and the second reconstructed model around a reference keypoint, which is one of the multiple keypoints.

Images