Method, system, device and storage medium for reconstructing three-dimensional dynamic model of object

Abstract

The present disclosure relates to a method, system, device and storage medium for reconstructing a three-dimensional dynamic model of an object. The method includes: creating a three-dimensional structural model suitable for an object in a static state; creating a posture model of the object in a dynamic state; for a limb segment, creating a corresponding virtual skeleton segment in the three-dimensional structural model, the virtual skeleton segment having a starting endpoint and an end endpoint; for each frame, transforming the three-dimensional structural model to align it with the posture model; for each frame, making the third vector from the starting endpoint to the end endpoint of the virtual skeleton segment and the fourth vector from the first coordinate point to the second coordinate point in the posture model collinear, obtaining a virtual three-dimensional model of the object in each frame; and combining the virtual three-dimensional models of each frame in the time period to obtain a three-dimensional dynamic model. This can achieve rapid three-dimensional dynamic modeling of the object.

Description

Technical Field

[0001] The present disclosure relates to the field of animal behavior and machine vision technology, and more specifically to a method, system, device and storage medium for reconstructing a three-dimensional dynamic model of an object. Background Art

[0002] Modeling the behavior of individual objects (animals or artificial agents) has great application value in both art and science. Accurately reconstructing the dynamic behavior process of an individual in real time can be used to study the sensorimotor transformation of an individual in science, and can provide realistic visual dynamic materials in the field of film, television and games.

[0003] To achieve the reconstruction of dynamic individuals, existing technologies basically rely on manual modeling and motion capture systems. The labor and time costs of manual modeling and manual animation are high, and the degree of restoration depends on people's subjective realistic ability, making it difficult to achieve three-dimensional dynamic reconstruction of complex interactive scenes and long-term. The motion capture system used in the film and television and game fields is limited to the three-dimensional motion reconstruction of human and humanoid bodies because its original design intention is to capture human movements. It is difficult to apply to scenes of different scales, different forms, and multiple individuals interacting, especially the interaction process of small animals such as insects.

[0004] Current technology is able to obtain an individual's three-dimensional structural model (three-dimensional imaging technology) and posture model (three-dimensional key point detection technology), but how to use the posture model to drive the three-dimensional structural model (the two models are often not measured on the same individual, so they need to be matched) is related to the final presentation effect and requires an accurate and effective matching method. Summary of the invention

[0005] To solve the above problems, the present disclosure provides a method, system, device and storage medium for reconstructing a three-dimensional dynamic model of an object, which can achieve matching of a three-dimensional structure model and a posture model, thereby realizing fast and dynamic three-dimensional dynamic modeling of interactive objects (especially small objects).

[0006] The first scheme of the present disclosure provides a method for reconstructing a three-dimensional dynamic model of an object, comprising: creating a three-dimensional structural model suitable for the object in a static state, wherein the outer surface of the three-dimensional structural model is composed of a large number of polygons, and the polygons are surrounded by multiple vertices. The reconstruction method further comprises: creating a posture model of the object in a dynamic state, wherein the posture model indicates a spatiotemporal sequence of three-dimensional coordinates of key points of the object's limbs within a specified time period. The reconstruction method further comprises: for each limb segment that can move independently, creating a corresponding virtual skeleton segment in the three-dimensional structural model, and specifying the bone weight of the virtual skeleton segment relative to each vertex of the multiple vertices, wherein the virtual skeleton segment has a starting endpoint and an end endpoint, and the bone weight represents the degree of consistency between the movement of the virtual skeleton segment and the movement of the vertex. The reconstruction method further includes: for each frame in the time period, transforming the three-dimensional structure model to align it with the posture model, including: translating the three-dimensional structure model as a whole so that its first center coincides with the second center of the posture model, wherein the relative position of the first center in the three-dimensional structure model corresponds to the relative position of the second center in the posture model; rotating the three-dimensional structure model as a whole around the first center so that the direction of the first vector of the three-dimensional structure model is consistent with the direction of the second vector of the posture model, wherein the first vector represents the vector between the endpoints of at least one virtual skeleton segment, the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segment, and the first vector and the second vector correspond to the same body region that can represent the overall direction of the body. The reconstruction method further includes: for each frame in the time period, adjusting each virtual skeleton segment in the three-dimensional structure model so that the third vector from the starting endpoint to the end endpoint of the virtual skeleton segment and the fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining the virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint, and the second coordinate point is associated with the end endpoint. The reconstruction method further comprises: combining the virtual three-dimensional models of the frames within the time period to obtain the three-dimensional dynamic model of the object.

[0007] A second solution of the present disclosure provides a system for reconstructing a three-dimensional dynamic model of an object, which includes at least one processor configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first solution.

[0008] A third solution of the present disclosure provides a device for reconstructing a three-dimensional dynamic model of an object, which includes at least one processor configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first solution.

[0009] A fourth aspect of the present disclosure provides a storage medium storing a computer program for causing a computer to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. When appropriate, the same reference numerals are used in all drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present device or method. In the drawings:

[0011] Figure 1 is a schematic flow chart of a method for reconstructing a three-dimensional dynamic model of an object according to an embodiment of the present disclosure;

[0012] Figure 2a-2c A schematic diagram of creating a three-dimensional structure model of a fruit fly according to an embodiment of the present disclosure;

[0013] Figure 3a-3c is a schematic diagram of a posture model of a fruit fly according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a three-dimensional structural model of a fruit fly according to an embodiment of the present disclosure, showing virtual bone segments created for limb segments;

[0015] Figure 5 is a schematic diagram of a three-dimensional structural model of a fruit fly according to an embodiment of the present disclosure, and shows, with different shades of color, the weights of the vertices on the three-dimensional surface assigned to the virtual skeleton segments;

[0016] Figure 6 A schematic flow chart of a method for transforming a three-dimensional structure model to align it with a posture model is shown;

[0017] Figure 7a-7d is a schematic diagram of moving, rotating and scaling a virtual skeleton segment according to an embodiment of the present disclosure;

[0018] Figure 8a and Figure 8b are diagrams showing a plurality of virtual skeleton segments and corresponding coordinate points according to an embodiment of the present disclosure;

[0019] Fig. 9 A virtual scene composed of multiple virtual three-dimensional models of fruit flies constructed according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0021] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of well-known functions and well-known components.

[0023] Figure 1 A schematic flow chart of a method for reconstructing a three-dimensional dynamic model of an object in a dynamic interaction scene is shown. The object herein may include any one of an animal, a movable facility, and a robot. That is, the reconstruction method provided by the present disclosure is not only suitable for conventional large animals, large movable facilities, and large robots in the prior art, but is also preferably suitable for small animals, small movable facilities, and small robots. The so-called "small" here means that the scale is relatively small relative to the shooting field of view. For example, small animals may include but are not limited to insects, small fish, small amphibians, small reptiles, etc. For ease of description, the reconstruction method of the three-dimensional dynamic model of the object provided by the present disclosure is described in detail below using the classic model animal fruit fly as an example. However, it should be understood that the method is also applicable to the model reconstruction of other above-mentioned objects other than fruit flies.

[0024] like Figure 1 As shown, the reconstruction method 100 may include the following steps.

[0025] In step S110, a three-dimensional structural model suitable for the object in a static state is created, wherein an outer surface of the three-dimensional structural model is composed of a plurality of polygons, and the polygons are surrounded by a plurality of vertices.

[0026] It is well known in the art that the three-dimensional structure model of an object can represent the three-dimensional structure and surface morphology of the object in a static state. First, a known three-dimensional imaging technology (such as X-ray imaging, ultrasound imaging, etc.) is used to perform a tomographic scan on the processed static fruit fly sample to obtain the following image: Figure 2a Then, the existing 3D reconstruction technology can be used to obtain a static 3D model, and the static 3D model can be edited to retain only the outermost surface and repair the surface damaged during the reconstruction process to obtain the following Figure 2b Finally, the camera is used to take detailed images of various parts of the static fruit fly, and the texture and material of the repaired static three-dimensional model are produced with reference to the image, and applied to the static three-dimensional model to obtain the following Figure 2c The image shown can most completely and accurately reflect the three-dimensional structure and surface morphology of the fruit fly.

[0027] Here, the three-dimensional structural model is obtained based on three-dimensional imaging technology, which greatly improves the accuracy and speed of interactive scene modeling compared to traditional manual modeling methods.

[0028] Because fruit flies have similar surface morphology and physiological structures, Figure 2c The three-dimensional structural model shown can be applied to the reconstruction of other fruit fly individuals.

[0029] In addition, in the creation of a three-dimensional dynamic model of the fruit fly, Figure 2c The outer surface of the three-dimensional structure model shown is composed of a plurality of adjacent polygons, each of which is surrounded by a plurality of vertices, such as Figure 4 As shown, the polygon here can be a triangle, a quadrilateral, a pentagon, etc. In this way, the outer surface of the fruit fly is distributed with multiple vertices, each vertex is the intersection of the edges of two or more adjacent polygons, and each vertex can be driven by multiple adjacent bones based on pre-assigned bone weights.

[0030] Next, in step S120, a posture model of the object in a dynamic state is created.

[0031] As is also known in the art, the posture model indicates a spatiotemporal sequence of three-dimensional coordinates of key points of the subject's limbs within a specified period of time.

[0032] Specifically, first, the existing three-dimensional key point detection method (such as manual annotation, body surface marker recognition, deep neural network, etc.) is used to identify the key points of the limbs of the fruit fly, and the three-dimensional coordinates of each limb in a specified time period are obtained through key point tracking technology, and stored in a separate storage device. The set of three-dimensional coordinates of each frame of each limb key point in the specified time period constitutes the posture model of the fruit fly in the specified time period. It should be understood that the "limbs" here include but are not limited to the head, face, neck, shoulders, chest, abdomen, waist, left / right forelimbs, left / right middle limbs, left / right hind limbs, wings, tail and antennae, etc. Figure 3a-3c All key points of the fruit fly at three moments (frames) are exemplarily shown. These three moments correspond to the action behaviors of the fruit fly, and the three-dimensional coordinates of these key points are stored in the storage device.

[0033] Generally speaking, each limb of the subject includes at least one independently movable limb segment (e.g. Figure 4 As shown, the right hind limb of the fruit fly can be composed of 4 independently movable limb segments S1-S4), and the joint positions between adjacent limb segments are usually identified as corresponding key points. In this way, the movement of the object can be more accurately identified and the accuracy of three-dimensional modeling can be improved. However, auxiliary key points can also be set in the middle position of each limb segment itself, which will not be described in detail in this embodiment.

[0034] In step S130, for each limb segment that can move independently, a corresponding virtual bone segment is created in the three-dimensional structure model, and a bone weight of the virtual bone segment relative to each vertex among the multiple vertices is specified, and the bone weight represents the degree of consistency between the movement of the virtual bone segment and the movement of the vertex.

[0035] As described above, each limb of the object may include at least one limb segment, and these limb segments can move independently. Therefore, it is necessary to create corresponding virtual bone segments for each limb segment in the three-dimensional structural model of the object created in step S110. This can be achieved by using, for example, Blender software. Figure 4 As shown, the right hind limb of the fruit fly has four independently movable limb segments S1-S4. Correspondingly, a corresponding virtual skeleton segment can be created for each of the four limb segments, wherein the virtual skeleton segment has a starting endpoint and an end endpoint, and these virtual skeleton segments are sequentially connected end to end through joints to form a skeleton corresponding to the right hind limb. In this way, a corresponding virtual skeleton segment can be created for each limb segment of each limb of the fruit fly.

[0036] As described above, in the posture model of the fruit fly, the joint positions between adjacent limb segments are preferably identified as corresponding key points. Correspondingly, a virtual skeleton segment corresponding to the limb segment is preferably created between the joints at the endpoints of the limb segment, and its starting endpoint and end endpoint both correspond to the key points at the joint positions (the first key point and the second key point described later).

[0037] Since the movement of each virtual bone segment will drive the movement of at least a part of the outer surface of the object, in order to establish the connection between the movement of the virtual bone segment and the movement of each vertex on the outer surface of the three-dimensional structure model (i.e., define the rules for the movement of the limbs to drive the movement of the three-dimensional surface), a bone weight can be assigned to each virtual bone segment. Here, the "bone weight" indicates the consistency between the movement of the virtual bone segment and the movement of the vertex. This can also be achieved through Blender software.

[0038] Specifically, Figure 5 As shown, for each of all the virtual bone segments of the object, such as the virtual bone segment corresponding to segment S2, the bone weight of each vertex on the outer surface of the three-dimensional structure model can be pre-set in the Blender software and represented by color depth in the figure. The darker the color, the greater the weight, and the closer the movement of the vertex is to the movement of the virtual bone segment.

[0039] Then, in step S140, for each frame within the specified time period, the three-dimensional structure model is transformed so as to align it with the posture model. Figure 6 As shown, step S140 includes the following steps.

[0040] S141, translating the entire three-dimensional structure model so that a first center thereof coincides with a second center of the posture model, wherein a relative position of the first center in the three-dimensional structure model corresponds to a relative position of the second center in the posture model.

[0041] S142, rotate the three-dimensional structure model as a whole around the first center so that the direction of the first vector of the three-dimensional structure model is consistent with the direction of the second vector of the posture model, wherein the first vector represents the vector between the endpoints of at least one of the virtual skeleton segments, the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segments, and the first vector and the second vector correspond to the same body area that can express the overall direction of the body.

[0042] In this step, in order to obtain the virtual three-dimensional model of the object in each frame, the three-dimensional structure model in the static posture obtained in step S130 and the posture model of the frame obtained in step S120 can be bound. This step can be implemented by using Unity3D software using C# script.

[0043] First, in the current frame, the three-dimensional structure model obtained in step S130 is moved so that the point of its first center coincides with the point of the second center of the posture model of the frame obtained in step S120, wherein the first center and the second center can be pre-identified and determined in the corresponding model, and their three-dimensional coordinates are stored in a storage device. The relative position of the first center in the three-dimensional structure model should correspond to the relative position of the second center in the posture model. It should be understood that the second center in the posture model can also be one of the key points identified in step S120.

[0044] Then, for more accurate binding, the orientation of the object body axis represented by the three-dimensional structure model can be adjusted to be basically consistent with the orientation of the body axis represented by the posture model of the frame. Specifically, a first vector can be specified in the three-dimensional structure model, and the first vector represents the vector between the endpoints of at least one virtual skeleton segment, and then a second vector can be specified in the posture model, and the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segment, that is, the vector from the key point to the key point. The first vector and the second vector should correspond to the same body area that can express the overall direction of the body, for example, the first vector represents the vector from the end endpoint of the virtual skeleton segment of the tail to the front end endpoint of the virtual skeleton segment of the head of the body, and the corresponding second vector represents the vector from the key point at the tail to the key point at the head. Next, by adjusting the first vector and the second vector to be consistent in direction, the two models can be kept consistent in orientation (aligned) to the greatest extent, which provides a guarantee for the matching of the subsequent skeleton segments and coordinate points (described later).

[0045] Here, the first vector and the second vector should be vectors that can best represent the orientation of the corresponding model, and the starting point and the end point of the vector should be selected so that the possibility of axial bending of the limbs in the middle is minimized. As an example, preferably, since the body area from the waist to the neck generally has a relatively fixed orientation, deformations such as twisting and bending usually do not occur, so that the orientation of the body area can roughly represent the axial direction of the body, the first vector can be a vector from the end of the virtual bone segment corresponding to the waist of the fruit fly to the end of the corresponding virtual bone segment of the virtual bone segment corresponding to the neck, and the corresponding second vector can be a vector from the key point at the corresponding position of the waist of the fruit fly to the key point at the corresponding position of the neck.

[0046] For another example, alternatively, the first vector may also be a vector from the end of the virtual skeleton segment corresponding to the fruit fly's tail to the end of the virtual skeleton segment corresponding to the head, and the corresponding second vector may be a vector from the key point at the corresponding position of the end of the fruit fly's tail to the key point at the corresponding position of the head.

[0047] Next, in step S150, each of the virtual skeleton segments in the three-dimensional structure model is adjusted so that the third vector from the starting endpoint to the end endpoint of the virtual skeleton segment and the fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining a virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint, and the second coordinate point is associated with the end endpoint.

[0048] In this step, for the object, the three-dimensional key point coordinate data is used to drive the skeleton, and the dynamic process within the specified time period is processed frame by frame at the selected viewing angle to render a dynamic scene, where the input of each frame is the coordinates of all the three-dimensional key points corresponding to the object.

[0049] Specifically, Figure 7a-7d As shown, the adjustment of the virtual skeleton segment in step S150 includes changing its position.

[0050] - Adjustment of the position of the virtual skeleton segment

[0051] In the above step S130, for each limb segment, a corresponding virtual skeleton segment is created in the three-dimensional structure model. The virtual skeleton segment has a starting endpoint B1 and an end endpoint B2, such as Figure 7a In this embodiment, a third vector from B1 to B2 for representing the orientation of the virtual skeleton segment is specified in the three-dimensional structure model, and a fourth vector from a first coordinate point P1 associated with the starting endpoint B1 to a second coordinate point P2 associated with the end endpoint B2 is specified in the posture model.

[0052] By translating the starting endpoint B1 of the virtual skeleton segment to the position of the first coordinate point P1 (such as Figure 7b Then, the virtual skeleton segment is pivoted with the starting endpoint B1 as the pivot point (as shown in Figure 7c As shown), the direction of the third vector is consistent with the direction of the fourth vector, so that the third vector and the fourth vector are collinear.

[0053] In a preferred embodiment, as mentioned above, in the posture model of the fruit fly, the joint positions between adjacent limb segments are identified as corresponding key points, and a virtual skeleton segment corresponding to the limb segment is created between two adjacent key points corresponding to the limb segment, that is, the first coordinate point P1 and the second coordinate point P2 are both corresponding first and second key points at the corresponding positions of the virtual skeleton segment, and the three-dimensional coordinates of the coordinate points are the three-dimensional coordinates of the corresponding key points, wherein the starting endpoint B1 is moved to the first key point and the end endpoint B2 is rotated toward the second key point to complete the directional matching between the virtual skeleton segment in the three-dimensional structure model and the segment between the first and second key points in the posture model.

[0054] In another embodiment, if in step S130, the virtual skeleton segments Ba to Bx are created without corresponding key points corresponding to at least one of the starting endpoint Ba and the end endpoint Bx, that is, at least one of the first coordinate point and the second coordinate point is not any key point in the posture model, then the three-dimensional coordinates of the first coordinate point or the second coordinate point need to be calculated separately.

[0055] In this embodiment, a linear combination of the three-dimensional coordinates of multiple key points in the posture model near the position corresponding to the starting endpoint Ba can be selected to calculate the three-dimensional coordinates of the first coordinate point, and a linear combination of the three-dimensional coordinates of multiple key points in the posture model near the position corresponding to the end endpoint Bx can also be selected to calculate the three-dimensional coordinates of the second coordinate point. For example, Figure 8a and Figure 8b The figure shows the calculation process of the three-dimensional coordinates Px of the second coordinate point x when there is a key point Ka corresponding to the starting endpoint Ba but no key point corresponding to the ending endpoint Bx. Figure 8a The three-dimensional structure model in a static posture shows the three-segment skeleton Ba to Bb, Ba to Bc and Ba to Bx. Figure 8b The second coordinate point x and its surrounding three key points Ka, Kb and Kc (the three-dimensional coordinates of these three key points P Ka , P Kb , P Kc It can be read from the aforementioned storage device). Here, the key point Ka is exactly the key point corresponding to the starting endpoint Ba, and in an optional embodiment, another nearby key point can also be selected to replace the key point Ka. The linear relationship Bx=w1*Ba+w2*Bb+w3*Bc can be obtained by the position of the endpoints of each virtual skeleton segment in the static state. The three parameters (w1, w2, w3) are obtained by solving the linear equation group. Substituting the three-dimensional key point coordinates into P x =w1*P Ka +w2*P Kb +w3*P Kc .

[0056] Through the above process, it is possible to calculate the three-dimensional coordinates of the first coordinate point and the second coordinate point corresponding to the starting endpoint Ba and / or the end endpoint Bx respectively when there are no key points corresponding to the starting endpoint Ba and / or the end endpoint Bx in the posture model, and translate and pivot the virtual skeleton segment through the aforementioned process so that the third vector and the fourth vector are collinear.

[0057] It should be understood that in the process of pivoting the virtual skeleton segment to change its spatial position, it should be ensured that the skeleton segment does not rotate around its longitudinal axis, that is, no rotation occurs from the static state to the current state. If it is necessary to rotate the skeleton segment around its longitudinal axis, it is necessary to add an additional skeleton segment perpendicular to the virtual skeleton segment during the process of generating the virtual skeleton segment for each limb segment in step S130.

[0058] - Adjustment of the spatial shape of the virtual skeleton segment

[0059] After completing the above-mentioned vector collinearity, it is necessary to set the spatial shape of the virtual bone segment. The spatial shape here preferably refers to its length in the axial direction.

[0060] Specifically, for a retractable limb segment, such as a segment of the abdomen of a fruit fly, the length of the virtual bone segment can be scaled along the axial direction so that the size of the third vector (the length from the starting endpoint B1 to the end endpoint B2) is equal to the size of the fourth vector (the length from the first coordinate point P1 to the second coordinate point P2), that is, the third vector and the fourth vector are equal, as shown in FIG. Figure 7d As shown. For a non-retractable limb segment, such as a segment of a fruit fly wing, the length of the virtual bone segment can be fixedly set so that the size of the third vector (the length from the starting endpoint B1 to the end endpoint B2) is the average of the sizes of the fourth vector (the length from the first coordinate point P1 to the second coordinate point P2) in multiple frames, as shown Figure 7c As shown; in this case, the end point B2 of the virtual skeleton segment may not coincide with the second coordinate point P2, which can reduce the abnormal expansion and contraction of the limb segment caused by the three-dimensional key point detection error.

[0061] Optionally, in this embodiment, the adjustment of the spatial shape of the virtual skeleton segment also includes adjusting the width in a direction perpendicular to the axial direction, so that the virtual skeleton segment can be more accurately aligned with the corresponding key point, which is not described in detail in this specification.

[0062] According to the above steps S140-S150, a virtual three-dimensional model of the object in each frame can be obtained, in which, due to the aforementioned pre-set bone weights, the vertices on the three-dimensional surface will also move to corresponding positions as the bone segments move. Then, in step S160, the virtual three-dimensional models of each frame in the specified time period can be combined to obtain a three-dimensional dynamic model of the object in the specified time period.

[0063] After completing the reconstruction of the 3D dynamic model of a single object in the scene, the above process can be repeated for the remaining objects in the scene to obtain 3D dynamic models of multiple objects, that is, the entire dynamic interaction scene is reconstructed. Fig. 9 shown.

[0064] The reconstruction method of the three-dimensional dynamic model of an object provided in the present disclosure realizes the rapid three-dimensional dynamic modeling of the object by combining the three-dimensional structure model and the posture model. In addition, the posture data-driven three-dimensional model based on the three-dimensional key points indicated by the posture model does not rely on prior knowledge and can accurately and in real time reconstruct natural behavior. In the specific implementation technology, the three-dimensional key points are preferably corresponded to the endpoints of the skeleton, and the three-dimensional coordinates of the corresponding coordinate points are estimated using the nearby key points for the skeleton endpoints without key point correspondence, so that all the limbs corresponding to the skeleton can move to the specified position, effectively restoring the posture of the object. For the non-retractable (fixed length) limb segments, the average length is used as the skeletal length to avoid the abnormal extension and contraction of the limb segments caused by detection errors.

[0065] In addition, the present disclosure also provides a reconstruction system for a three-dimensional dynamic model of an object, the reconstruction system comprising at least one processor, and the at least one processor can be configured to execute a reconstruction method 100 of a three-dimensional dynamic model according to an embodiment of the present disclosure. Various examples and details of the above methods can be selectively combined here and are not repeated here.

[0066] Similarly, the present disclosure also provides a reconstruction device for a three-dimensional dynamic model of an object, the reconstruction device comprising at least one processor, and the at least one processor can be configured to execute a reconstruction method 100 of a three-dimensional dynamic model according to an embodiment of the present disclosure. Various examples and details of the above methods can be selectively combined here and are not repeated here.

[0067] By using the reconstruction system and the reconstruction device, the beneficial effects described in conjunction with the reconstruction method 100 can also be achieved, including achieving rapid and accurate reconstruction of the three-dimensional dynamic model of the object.

[0068] The present disclosure may also be implemented by providing a computer program that implements the functions described in the above embodiments to a computer and enabling one or more processors of the computer to read and execute the program. In various embodiments of the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It should be noted that the processor may also be integrated with components for storage such as memory units and / or cache units.

[0069] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in a processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0070] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0071] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0072] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0073] Multiple embodiments of the present disclosure are described in detail above, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should all fall within the scope of protection required by the present disclosure.

Claims

1. A method for reconstructing a three-dimensional dynamic model of an object, characterized in that: include: Creating a three-dimensional structural model suitable for the object in a static state, wherein the outer surface of the three-dimensional structural model is composed of a large number of polygons, and the polygons are surrounded by a plurality of vertices; Creating a posture model of the object in a dynamic state, the posture model indicating a spatiotemporal sequence of three-dimensional coordinates of key points of the object's limbs within a specified time period; For each limb segment capable of independent movement, a corresponding virtual bone segment is created in the three-dimensional structure model, and a bone weight of the virtual bone segment relative to each vertex of the plurality of vertices is specified, wherein the virtual bone segment has a starting endpoint and an end endpoint, and the bone weight indicates the consistency of the movement of the virtual bone segment with the movement of the vertex; For each frame in the time period, transforming the three-dimensional structure model so as to align it with the posture model includes: Translating the entire three-dimensional structure model so that a first center thereof coincides with a second center of the posture model, wherein a relative position of the first center in the three-dimensional structure model corresponds to a relative position of the second center in the posture model; Rotate the three-dimensional structure model as a whole around the first center so that the first vector of the three-dimensional structure model is consistent in direction with the second vector of the posture model, wherein the first vector represents a vector between endpoints of at least one of the virtual skeleton segments, the second vector represents a vector between key points corresponding to the endpoints of the virtual skeleton segments, and the first vector and the second vector correspond to the same body region capable of expressing the overall direction of the body; For each frame in the time period, adjusting each of the virtual skeleton segments in the three-dimensional structure model so that a third vector from the starting endpoint to the end endpoint of the virtual skeleton segment and a fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining a virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint, and the second coordinate point is associated with the end endpoint; and The virtual three-dimensional models of the frames within the time period are combined to obtain the three-dimensional dynamic model of the object.

2. The reconstruction method according to claim 1, characterized in that: Adjusting each of the virtual bone segments in the three-dimensional structure model includes: The position of the virtual skeletal segment is changed.

3. The reconstruction method according to claim 2, characterized in that: Changing the position of the virtual skeleton segment comprises: Translating the virtual skeleton segment so that the starting endpoint coincides with the first coordinate point; The virtual skeleton segment is pivoted with the starting endpoint as a pivot point so that the third vector and the fourth vector are collinear.

4. The reconstruction method according to claim 2 or 3, characterized in that: Adjusting each of the virtual bone segments in the three-dimensional structure model further includes: The spatial shape of the virtual skeletal segment is changed.

5. The reconstruction method according to claim 4, characterized in that: Changing the spatial shape of the virtual skeleton segment comprises: The length of the virtual bone segment is adjusted.

6. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional coordinates of the first coordinate point are the three-dimensional coordinates of the first key point in the posture model corresponding to the starting endpoint, and the three-dimensional coordinates of the second coordinate point are the three-dimensional coordinates of the second key point in the posture model corresponding to the ending endpoint.

7. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional coordinates of the first coordinate point are calculated based on the coordinates of multiple key points in the posture model near the position corresponding to the starting endpoint, and / or the three-dimensional coordinates of the second coordinate point are calculated based on the coordinates of multiple key points in the posture model near the position corresponding to the end endpoint.

8. The reconstruction method according to claim 5, characterized in that: For the retractable limb segment, the length of the virtual bone segment is axially scaled so that the magnitude of the third vector is equal to the magnitude of the fourth vector.

9. The reconstruction method according to claim 5, characterized in that: For the non-stretchable limb segment, the length of the virtual skeleton segment is set so that the magnitude of the third vector is an average value of the magnitudes of the fourth vector in a plurality of frames.

10. The reconstruction method according to any one of claims 1 to 3, characterized in that: The first vector represents a vector from a terminal end point of a virtual skeleton segment at the tail of the body to a front end point of a virtual skeleton segment at the head of the body, and the second vector represents a vector from a key point at the tail to a key point at the head.

11. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional structure model is obtained by performing tomographic scanning on a sample of the processed static object using a three-dimensional imaging technology.

12. The reconstruction method according to any one of claims 1 to 3, characterized in that: The posture model is obtained by acquiring the three-dimensional coordinates of the key points of each limb of all frames in the time period using a three-dimensional key point detection method.

13. The reconstruction method according to any one of claims 1 to 3, characterized in that: The object includes any one of an animal, a movable facility, and a robot.

14. A system for reconstructing a three-dimensional dynamic model of an object, characterized in that: include: At least one processor is configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to any one of claims 1 to 13. 15 . A storage medium storing a computer program for causing a computer to execute the method for reconstructing a three-dimensional dynamic model of an object according to claim 1 .

Images