Reconstruction method, system and apparatus for three-dimensional dynamic model of object, and storage medium

Abstract

The present disclosure relates to a reconstruction method, system and apparatus for a three-dimensional dynamic model of an object, and a storage medium. The method comprises: creating a three-dimensional structural model suitable for a static state of an object; creating a pose model of the object in a dynamic state; for limb sections, creating corresponding virtual skeleton sections in the three-dimensional structural model, wherein each virtual skeleton section has a start end point and a tail end point; for each frame, transforming the three-dimensional structural model, so as to make same align with the pose model; for each frame, making a third vector of each virtual skeleton section from the start end point to the tail end point collinear with a fourth vector from a first coordinate point to a second coordinate point in the pose model, so as to obtain virtual three-dimensional models of the object in the frames; and combining the virtual three-dimensional models in the frames within a time period, so as to obtain a three-dimensional dynamic model. Therefore, fast three-dimensional dynamic modeling of objects can be implemented.

Description

Method, system, device and storage medium for reconstructing three-dimensional dynamic model of object Technical Field

[0001] The present disclosure relates to the fields of animal behavior and machine vision technology, and more particularly to a method, system, device, and storage medium for reconstructing a three-dimensional dynamic model of an object. Background Art

[0002] Modeling the behavior of individual objects (animals or artificial agents) has enormous applications in both art and science. Accurately reconstructing an individual's dynamic behavior in real time can be used to study sensorimotor transformations in science and provide realistic visual dynamics in film, television, and gaming.

[0003] To reconstruct dynamic individuals, existing technologies rely primarily on manual modeling and motion capture systems. Manual modeling and hand-animation production are labor-intensive and time-consuming, and the degree of fidelity relies on the subjective ability to create realistic images, making it difficult to achieve three-dimensional dynamic reconstruction of complex interactive scenes and long-term scenarios. Motion capture systems used in film, television, and gaming, originally designed to capture human motion, are limited to reconstructing the three-dimensional motion of humans and humanoids. This makes them difficult to apply to scenarios involving multiple individuals of varying scales and forms, particularly those involving small animals like insects.

[0004] Current technology is capable of obtaining an individual's three-dimensional structural model (3D imaging technology) and posture model (3D key point detection technology). However, how to use the posture model to drive the three-dimensional structural model (the two models are often not measured on the same individual, so they need to be matched) is related to the final presentation effect and requires an accurate and effective matching method.

[0005] Summary of the Invention

[0006] To solve the above problems, the present disclosure provides a method, system, device and storage medium for reconstructing a three-dimensional dynamic model of an object, which can achieve matching of three-dimensional structure models and posture models, thereby realizing rapid dynamic three-dimensional dynamic modeling of interactive objects (especially small objects).

[0007] The first scheme of the present disclosure provides a method for reconstructing a three-dimensional dynamic model of an object, comprising: creating a three-dimensional structural model suitable for the object in a static state, wherein the outer surface of the three-dimensional structural model is composed of a large number of polygons, and the polygons are surrounded by a plurality of vertices. The reconstruction method further comprises: creating a posture model of the object in a dynamic state, wherein the posture model indicates a spatiotemporal sequence of three-dimensional coordinates of key points of the object's limbs within a specified time period. The reconstruction method further comprises: for each limb segment that can move independently, creating a corresponding virtual skeleton segment in the three-dimensional structural model, and specifying the bone weight of the virtual skeleton segment relative to each vertex of the multiple vertices, wherein the virtual skeleton segment has a starting endpoint and an end endpoint, and the bone weight represents the degree of consistency between the movement of the virtual skeleton segment and the movement of the vertex. The reconstruction method further includes: for each frame within the time period, transforming the three-dimensional structural model so that it is aligned with the posture model, including: translating the entire three-dimensional structural model so that its first center coincides with the second center of the posture model, wherein the relative position of the first center in the three-dimensional structural model corresponds to the relative position of the second center in the posture model; and rotating the entire three-dimensional structural model around the first center so that the direction of a first vector of the three-dimensional structural model is consistent with the direction of a second vector of the posture model, wherein the first vector represents the vector between the endpoints of at least one virtual skeleton segment, the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segment, and the first vector and the second vector correspond to the same body region capable of representing the overall direction of the body. The reconstruction method further includes: for each frame within the time period, adjusting each virtual skeleton segment in the three-dimensional structural model so that a third vector from the starting endpoint to the ending endpoint of the virtual skeleton segment and a fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining a virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint and the second coordinate point is associated with the ending endpoint. The reconstruction method further includes: combining the virtual three-dimensional models of the frames within the time period to obtain the three-dimensional dynamic model of the object.

[0008] A second aspect of the present disclosure provides a system for reconstructing a three-dimensional dynamic model of an object, comprising at least one processor configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first aspect.

[0009] A third aspect of the present disclosure provides a device for reconstructing a three-dimensional dynamic model of an object, comprising at least one processor configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first aspect.

[0010] A fourth aspect of the present disclosure provides a storage medium storing a computer program for causing a computer to execute the method for reconstructing a three-dimensional dynamic model of an object according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present disclosure. When appropriate, the same reference numerals are used in all drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present device or method. In the drawings:

[0012] FIG1 is a schematic flowchart of a method for reconstructing a three-dimensional dynamic model of an object according to an embodiment of the present disclosure;

[0013] 2a-2c are schematic diagrams of creating a three-dimensional structural model of a fruit fly according to an embodiment of the present disclosure;

[0014] 3a-3c are schematic diagrams of a posture model of a fruit fly according to an embodiment of the present disclosure;

[0015] FIG4 is a schematic diagram of a three-dimensional structural model of a fruit fly according to an embodiment of the present disclosure, showing virtual skeleton segments created for limb segments;

[0016] FIG5 is a schematic diagram of a three-dimensional structural model of a fruit fly according to an embodiment of the present disclosure, showing weights assigned to vertices on a three-dimensional surface for virtual skeletal segments in different shades of color;

[0017] FIG6 shows a schematic flow chart of a method for transforming a three-dimensional structure model to align it with a posture model;

[0018] 7a-7d are schematic diagrams of moving, rotating and scaling virtual skeleton segments according to an embodiment of the present disclosure;

[0019] 8a and 8b are diagrams respectively showing a plurality of virtual skeleton segments and corresponding coordinate points according to an embodiment of the present disclosure;

[0020] FIG9 is a virtual scene composed of multiple virtual three-dimensional models of fruit flies constructed according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0022] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0023] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components.

[0024] Figure 1 shows a schematic flow chart of a method for reconstructing a three-dimensional dynamic model of an object in a dynamic interactive scene. The objects herein may include any one of animals, movable facilities, and robots. That is, the reconstruction method provided by the present disclosure is not only suitable for conventional large animals, large movable facilities, and large robots in the prior art, but is also preferably suitable for small animals, small movable facilities, and small robots. The so-called "small" here means that the scale is relatively small relative to the shooting field of view. For example, small animals may include but are not limited to insects, small fish, small amphibians, small reptiles, etc. For ease of description, the reconstruction method of the three-dimensional dynamic model of the object provided by the present disclosure is described in detail below using the classic model animal fruit fly as an example. However, it should be understood that this method is also applicable to the reconstruction of models of other objects mentioned above other than fruit flies.

[0025] As shown in FIG1 , the reconstruction method 100 may include the following steps.

[0026] In step S110 , a three-dimensional structural model suitable for the object in a static state is created, wherein an outer surface of the three-dimensional structural model is composed of a plurality of polygons, and the polygons are surrounded by a plurality of vertices.

[0027] It is well known in the art that a three-dimensional structural model of an object can represent the object's three-dimensional structure and surface topography in a static state. First, a tomographic scan of a processed, static fruit fly sample is performed using known three-dimensional imaging techniques (such as X-ray imaging and ultrasound imaging), resulting in the image shown in Figure 2a. Next, a static three-dimensional model can be obtained using existing three-dimensional reconstruction techniques. This static three-dimensional model is then edited, retaining only the outermost surface layer and repairing any damaged surfaces during the reconstruction process, resulting in the image shown in Figure 2b. Finally, a camera is used to capture detailed images of various parts of the static fruit fly. Textures and materials for the repaired static three-dimensional model are created based on these images and applied to the static three-dimensional model, resulting in the image shown in Figure 2c, which most completely and accurately reflects the fruit fly's three-dimensional structure and surface topography.

[0028] Here, the three-dimensional structural model is obtained based on three-dimensional imaging technology, which greatly improves the accuracy and speed of interactive scene modeling compared to traditional manual modeling methods.

[0029] Since fruit flies have similar surface morphology and physiological structure, the three-dimensional structural model shown in Figure 2c can be applied to the reconstruction of other fruit fly individuals.

[0030] Furthermore, in creating a 3D dynamic model of the fruit fly, the outer surface of the 3D structural model shown in Figure 2c is composed of multiple adjacent polygons, each of which is surrounded by multiple vertices, as shown in Figure 4. The polygons here can be triangles, quadrilaterals, pentagons, etc. In this way, the outer surface of the fruit fly is distributed with multiple vertices, each vertex being the intersection of the edges of two or more adjacent polygons, and each vertex can be driven by multiple adjacent bones based on pre-assigned bone weights.

[0031] Next, in step S120 , a posture model of the object in a dynamic state is created.

[0032] As is also known in the art, the posture model indicates a spatiotemporal sequence of three-dimensional coordinates of key points of the subject's limbs within a specified time period.

[0033] Specifically, first, the limb key points of the fruit fly are identified using an existing three-dimensional key point detection method (such as manual annotation, surface marker recognition, deep neural network, etc.), and the three-dimensional coordinates of each limb in a specified time period are obtained through key point tracking technology, and stored in a separate storage device. The set of three-dimensional coordinates of each frame of each limb key point in the specified time period constitutes the posture model of the fruit fly in the specified time period. It should be understood that the "limbs" here include but are not limited to the head, face, neck, shoulders, chest, abdomen, waist, left / right forelimbs, left / right middle limbs, left / right hind limbs, wings, tail and antennae, etc. Figures 3a-3c exemplarily illustrate all the key points of the fruit fly at three moments (frames), and these three moments correspond to the movement behavior of the fruit fly, and the three-dimensional coordinates of these key points are all stored in the storage device.

[0034] Generally speaking, each limb of the object includes at least one independently movable limb segment (as shown in Figure 4, the right hind limb of the fruit fly can be composed of 4 independently movable limb segments S1-S4), and the joint positions between adjacent limb segments are usually identified as corresponding key points. In this way, the movement of the object can be more accurately identified and the accuracy of three-dimensional modeling can be improved. However, auxiliary key points can also be set in the middle position of each limb segment itself, which will not be described in detail in this embodiment.

[0035] In step S130, for each limb segment that can move independently, a corresponding virtual bone segment is created in the three-dimensional structural model, and a bone weight of the virtual bone segment relative to each vertex among the multiple vertices is specified. The bone weight represents the degree of consistency between the movement of the virtual bone segment and the movement of the vertex.

[0036] As described above, each limb of the object can include at least one limb segment, and these limb segments can move independently. Therefore, it is necessary to create corresponding virtual skeleton segments for each of its limb segments in the three-dimensional structural model of the object created in step S110. This can be achieved by using, for example, Blender software. As shown in Figure 4, the right hind limb of the fruit fly has 4 sections of limb segments S1-S4 that can move independently. Correspondingly, a corresponding virtual skeleton segment can be created for each segment in the 4 limb segments, wherein the virtual skeleton segment has a starting endpoint and an end endpoint, and these virtual skeleton segments are connected end to end through joints in turn to form a skeleton corresponding to its right hind limb. In this way, corresponding virtual skeleton segments can be created for each limb segment of each limb of the fruit fly.

[0037] As described above, in the Drosophila posture model, the joint positions between adjacent limb segments are preferably identified as corresponding key points. Accordingly, a virtual skeleton segment corresponding to the limb segment is preferably created between the joints at the endpoints of the limb segment, with the starting and ending endpoints of the virtual skeleton segment corresponding to the key points at the joint positions (the first key point and the second key point described below).

[0038] Because the movement of each virtual skeletal segment drives the movement of at least a portion of the object's outer surface, in order to establish a connection between the movement of the virtual skeletal segment and the movement of each vertex on the outer surface of the 3D structural model (i.e., to define the rules by which limb movement drives the movement of the 3D surface), each virtual skeletal segment can be assigned a bone weight. Here, a "bone weight" indicates the degree of consistency between the movement of the virtual skeletal segment and the movement of the vertex. This can also be achieved using Blender software.

[0039] Specifically, as shown in Figure 5, for each of all the virtual bone segments of the object, such as the virtual bone segment corresponding to segment S2, the bone weight of each vertex on the outer surface of the three-dimensional structure model can be pre-set in the Blender software and represented by light and dark colors in the figure. The darker the color, the greater the weight, and the closer the movement of the vertex is to the movement of the virtual bone segment.

[0040] Then, in step S140, for each frame within the specified time period, the three-dimensional structure model is transformed so as to be aligned with the posture model. Specifically, as shown in FIG6 , step S140 includes the following steps.

[0041] S141 , translating the entire three-dimensional structure model so that a first center thereof coincides with a second center of the posture model, wherein a relative position of the first center in the three-dimensional structure model corresponds to a relative position of the second center in the posture model.

[0042] S142, rotate the three-dimensional structure model as a whole around the first center so that the direction of the first vector of the three-dimensional structure model is consistent with the direction of the second vector of the posture model, wherein the first vector represents the vector between the endpoints of at least one of the virtual skeleton segments, and the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segments, and the first vector and the second vector correspond to the same body area that can express the overall direction of the body.

[0043] In this step, to obtain a virtual 3D model of the object in each frame, the 3D structure model in the static posture obtained in step S130 and the posture model of the frame obtained in step S120 can be bound. This step can be implemented using Unity3D software using C# scripts.

[0044] First, in the current frame, the three-dimensional structure model obtained in step S130 is moved so that the point of its first center coincides with the point of the second center of the posture model of the frame obtained in step S120, wherein the first center and the second center can be pre-identified and determined in the corresponding model, and their three-dimensional coordinates are stored in a storage device. The relative position of the first center in the three-dimensional structure model should correspond to the relative position of the second center in the posture model. It should be understood that the second center in the posture model can also be one of the key points identified in step S120.

[0045] Then, for more accurate binding, the orientation of the object body axis represented by the three-dimensional structure model can be adjusted to be basically consistent with the orientation of the body axis represented by the posture model of the frame. Specifically, a first vector can be specified in the three-dimensional structure model, and the first vector represents the vector between the endpoints of at least one virtual skeleton segment. Then, a second vector is specified in the posture model, and the second vector represents the vector between the key points corresponding to the endpoints of the virtual skeleton segment, that is, the vector from the key point to the key point. The first vector and the second vector should correspond to the same body region that can represent the overall direction of the body. For example, the first vector represents the vector from the end endpoint of the virtual skeleton segment at the tail of the body to the front end endpoint of the virtual skeleton segment at the head of the body, and the corresponding second vector represents the vector from the key point at the tail to the key point at the head. Next, by adjusting the first vector and the second vector to be consistent in direction, the two models can be kept in consistent orientation (alignment) to the greatest extent, providing a guarantee for the matching of subsequent skeleton segments and coordinate points (described later).

[0046] Here, the first vector and the second vector should be vectors that can best represent the orientation of the corresponding model, and the starting point and end point of the vectors should be selected so that the possibility of axial bending of the limbs in the middle is minimized. As an example, preferably, since the body region from the waist to the neck has a relatively fixed orientation and generally does not undergo deformations such as twisting and bending, the orientation of this body region can roughly represent the axial direction of the body. Therefore, the first vector can be a vector from the end of the virtual skeletal segment corresponding to the waist of the fruit fly to the end of the corresponding virtual skeletal segment of the neck, and the corresponding second vector can be a vector from the key point at the corresponding position of the fruit fly's waist to the key point at the corresponding position of the neck.

[0047] For another example, alternatively, the first vector may be a vector from the end of the virtual skeletal segment corresponding to the fruit fly's tail to the end of the virtual skeletal segment corresponding to the head, and the corresponding second vector may be a vector from the key point at the corresponding position of the fruit fly's tail end to the key point at the corresponding position of the head.

[0048] Next, in step S150, each of the virtual skeleton segments in the three-dimensional structure model is adjusted so that the third vector from the starting endpoint to the end endpoint of the virtual skeleton segment and the fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining a virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint, and the second coordinate point is associated with the end endpoint.

[0049] In this step, for the object, the three-dimensional key point coordinate data is used to drive the skeleton, and the dynamic process within the specified time period is processed frame by frame at the selected perspective to render a dynamic scene, where the input of each frame is the coordinates of all three-dimensional key points corresponding to the object.

[0050] Specifically, as shown in FIG. 7 a - FIG 7 d , the adjustment of the virtual skeleton segment in step S150 includes changing its position.

[0051] - Adjustment of the position of the virtual skeleton segment

[0052] In step S130, a corresponding virtual skeletal segment is created in the three-dimensional structural model for each limb segment. The virtual skeletal segment has a starting endpoint B1 and an end endpoint B2, as shown in FIG7a . In this embodiment, a third vector from B1 to B2 is defined within the three-dimensional structural model to represent the orientation of the virtual skeletal segment. A fourth vector from a first coordinate point P1 associated with the starting endpoint B1 to a second coordinate point P2 associated with the end endpoint B2 is defined within the posture model.

[0053] By translating the starting endpoint B1 of the above-mentioned virtual skeleton segment to the position of the above-mentioned first coordinate point P1 (as shown in Figure 7b), and then pivoting the virtual skeleton segment with the starting endpoint B1 as the pivot point (as shown in Figure 7c), the direction of the third vector is consistent with the direction of the fourth vector, so that the third vector and the fourth vector are collinear.

[0054] In a preferred embodiment, as mentioned above, in the posture model of the fruit fly, the joint positions between adjacent limb segments are identified as corresponding key points, and a virtual skeleton segment corresponding to the limb segment is created between two adjacent key points corresponding to the limb segment, that is, the first coordinate point P1 and the second coordinate point P2 are both the corresponding first and second key points at the corresponding positions of the virtual skeleton segment, and the three-dimensional coordinates of the coordinate points are the three-dimensional coordinates of the corresponding key points, wherein the starting endpoint B1 is moved to the first key point and the end endpoint B2 is rotated toward the second key point to complete the orientation matching between the virtual skeleton segment in the three-dimensional structure model and the segment between the first and second key points in the posture model.

[0055] In another embodiment, if in step S130, the virtual skeleton segments Ba to Bx are created without corresponding key points corresponding to at least one of the starting endpoint Ba and the end endpoint Bx, that is, at least one of the first coordinate point and the second coordinate point is not any key point in the posture model, then the three-dimensional coordinates of the first coordinate point or the second coordinate point need to be calculated separately.

[0056] In this embodiment, a linear combination of the three-dimensional coordinates of multiple key points in the posture model near the position corresponding to the starting endpoint Ba can be selected to calculate the three-dimensional coordinates of the first coordinate point, and a linear combination of the three-dimensional coordinates of multiple key points in the posture model near the position corresponding to the end endpoint Bx can also be selected to calculate the three-dimensional coordinates of the second coordinate point. For example, Figures 8a and 8b show the calculation process of the three-dimensional coordinates Px of the second coordinate point x when there is a key point Ka corresponding to the starting endpoint Ba, but no key point corresponding to the end endpoint Bx. Figure 8a shows the three segments of bones Ba to Bb, Ba to Bc and Ba to Bx given in the three-dimensional structure model in a static posture, and Figure 8b shows the three key points Ka, Kb and Kc of the second coordinate point x and its surroundings (the three-dimensional coordinates Px of these three key points Ka 、P Kb 、P Kc It can be read from the aforementioned storage device). Here, the key point Ka is exactly the key point corresponding to the starting endpoint Ba. In an optional embodiment, another nearby key point can also be selected to replace the key point Ka. The linear relationship Bx=w1*Ba+w2*Bb+w3*Bc can be obtained by the position of the endpoints of each virtual skeleton segment in the static state. The three parameters (w1, w2, w3) are obtained by solving the linear equation group. The three-dimensional key point coordinates are substituted into it to obtain P x =w1*P Ka +w2*P Kb +w3*P Kc .

[0057] Through the above process, it is possible to calculate the three-dimensional coordinates of the first coordinate point and the second coordinate point corresponding to the starting endpoint Ba and / or the end endpoint Bx respectively when there are no key points corresponding to the starting endpoint Ba and / or the end endpoint Bx in the posture model, and translate and pivot the virtual skeleton segment through the above process so that the third vector and the fourth vector are collinear.

[0058] It should be understood that when pivoting a virtual skeletal segment to change its spatial position, it should be ensured that the skeletal segment does not rotate about its longitudinal axis, that is, it does not rotate from the static state to the current state. If it is necessary to rotate the skeletal segment about its longitudinal axis, it is necessary to add an additional skeletal segment perpendicular to the virtual skeletal segment during the process of generating the virtual skeletal segment for each limb segment in step S130.

[0059] - Adjustment of the spatial shape of the virtual skeleton segment

[0060] After completing the above-mentioned vector collinearity, it is necessary to set the spatial shape of the virtual bone segment, where the spatial shape preferably refers to its length in the axial direction.

[0061] Specifically, for a retractable limb segment, such as the segment of the fruit fly's abdomen, the length of the virtual skeleton segment can be scaled along the axial direction so that the size of the third vector (the length from the starting endpoint B1 to the end endpoint B2) is equal to the size of the fourth vector (the length from the first coordinate point P1 to the second coordinate point P2), that is, the third vector and the fourth vector are equal, as shown in Figure 7d. For an inelastic limb segment, such as the segment of the fruit fly's wings, the length of the virtual skeleton segment can be fixedly set so that the size of the third vector (the length from the starting endpoint B1 to the end endpoint B2) is the average value of the size of the fourth vector (the length from the first coordinate point P1 to the second coordinate point P2) in multiple frames, as shown in Figure 7c; in this case, the end endpoint B2 of the virtual skeleton segment may not coincide with the second coordinate point P2, which can reduce the abnormal expansion and contraction of the limb segment caused by the three-dimensional key point detection error.

[0062] Optionally, in this embodiment, the adjustment of the spatial shape of the virtual skeleton segment also includes adjusting the width in a direction perpendicular to the axial direction so that the virtual skeleton segment can be more accurately aligned with the corresponding key point, which is not described in detail in this specification.

[0063] Following steps S140-S150, a virtual 3D model of the object is obtained for each frame. In this model, due to the pre-set bone weights, the vertices on the 3D surface move to corresponding positions as the bone segments move. Then, in step S160, the virtual 3D models of each frame within the specified time period are combined to obtain a dynamic 3D model of the object within the specified time period.

[0064] After completing the reconstruction of the 3D dynamic model of a single object in the scene, the above process can be repeated for the remaining objects in the scene to obtain 3D dynamic models of multiple objects, that is, the entire dynamic interaction scene is reconstructed, as shown in Figure 9.

[0065] The reconstruction method of the three-dimensional dynamic model of an object provided in the present disclosure realizes the rapid three-dimensional dynamic modeling of the object by combining the three-dimensional structure model and the posture model. In addition, the posture data of the three-dimensional key points indicated by the posture model drive the three-dimensional model, which does not rely on prior knowledge and can accurately and in real time reconstruct natural behavior. In terms of specific implementation technology, the three-dimensional key points are preferably mapped to the endpoints of the skeleton, and for the skeleton endpoints that do not have key points corresponding, the three-dimensional coordinates of the corresponding coordinate points are estimated using the key points nearby, so that the limbs corresponding to all bones can move to the specified position, effectively restoring the posture of the object. For non-retractable (fixed length) limb segments, the average length is used as the bone length to avoid the abnormal expansion and contraction of the limb segments due to detection errors.

[0066] In addition, the present disclosure further provides a system for reconstructing a three-dimensional dynamic model of an object, the system comprising at least one processor configured to execute a method 100 for reconstructing a three-dimensional dynamic model according to an embodiment of the present disclosure. Various examples and details of the above methods may be selectively incorporated herein and are not further elaborated upon.

[0067] Similarly, the present disclosure also provides a device for reconstructing a three-dimensional dynamic model of an object, the device comprising at least one processor configured to execute a method 100 for reconstructing a three-dimensional dynamic model according to an embodiment of the present disclosure. Various examples and details of the above methods may be selectively incorporated herein and are not further elaborated upon.

[0068] By utilizing the reconstruction system and the reconstruction device, the beneficial effects described above in conjunction with the reconstruction method 100 can also be achieved, including achieving rapid and accurate reconstruction of the three-dimensional dynamic model of the object.

[0069] The present disclosure may also be implemented by providing a computer program that implements the functions described in the above embodiments to a computer and enabling one or more processors of the computer to read and execute the program. In various embodiments of the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It should be noted that the processor may also be integrated with components for storage such as memory units and / or cache units.

[0070] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0071] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0072] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0073] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0074] The above describes in detail multiple embodiments of the present disclosure, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should fall within the scope of protection required by the present disclosure.

Claims

1. A method for reconstructing a three-dimensional dynamic model of an object, characterized in that: include: Creating a three-dimensional structural model suitable for the object in a static state, wherein an outer surface of the three-dimensional structural model is composed of a large number of polygons, and the polygons are surrounded by a plurality of vertices; creating a posture model of the object in a dynamic state, the posture model indicating a spatiotemporal sequence of three-dimensional coordinates of key points of the object's limbs within a specified time period; For each limb segment capable of independent movement, a corresponding virtual skeleton segment is created in the three-dimensional structural model, and a bone weight of the virtual skeleton segment relative to each of the plurality of vertices is specified, wherein the virtual skeleton segment has a start endpoint and an end endpoint, and the bone weight represents the degree of consistency between the movement of the virtual skeleton segment and the movement of the vertex; For each frame within the time period, transforming the three-dimensional structure model to align it with the posture model, including: translating the entire three-dimensional structural model so that a first center thereof coincides with a second center of the posture model, wherein a relative position of the first center in the three-dimensional structural model corresponds to a relative position of the second center in the posture model; Rotating the entire three-dimensional structural model around the first center so that a first vector of the three-dimensional structural model is aligned in direction with a second vector of the posture model, wherein the first vector represents a vector between endpoints of at least one virtual skeleton segment, the second vector represents a vector between key points corresponding to the endpoints of the virtual skeleton segment, and the first vector and the second vector correspond to the same body region capable of representing the overall direction of the body; For each frame within the time period, adjusting each virtual skeleton segment in the three-dimensional structure model so that a third vector from the starting endpoint to the ending endpoint of the virtual skeleton segment and a fourth vector from the first coordinate point to the second coordinate point in the posture model are collinear, thereby obtaining a virtual three-dimensional model of the object in each frame, wherein the first coordinate point is associated with the starting endpoint and the second coordinate point is associated with the ending endpoint; and The virtual three-dimensional models of the frames within the time period are combined to obtain the three-dimensional dynamic model of the object.

2. The reconstruction method according to claim 1, characterized in that Adjusting each of the virtual bone segments in the three-dimensional structure model includes: The position of the virtual skeletal segment is changed.

3. The reconstruction method according to claim 2, characterized in that: Changing the position of the virtual skeletal segment includes: Translating the virtual skeleton segment so that the starting endpoint coincides with the first coordinate point; The virtual skeleton segment is pivoted with the starting endpoint as a pivot point so that the third vector and the fourth vector are collinear.

4. The reconstruction method according to claim 2 or 3, characterized in that: Adjusting each of the virtual bone segments in the three-dimensional structure model further includes: The spatial shape of the virtual skeletal segment is changed.

5. The reconstruction method according to claim 4, characterized in that: Changing the spatial shape of the virtual skeletal segment comprises: The length of the virtual bone segment is adjusted.

6. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional coordinates of the first coordinate point are the three-dimensional coordinates of the first key point corresponding to the starting endpoint in the posture model, and the three-dimensional coordinates of the second coordinate point are the three-dimensional coordinates of the second key point corresponding to the ending endpoint in the posture model.

7. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional coordinates of the first coordinate point are calculated based on the coordinates of multiple key points in the posture model near the position corresponding to the starting endpoint, and / or the three-dimensional coordinates of the second coordinate point are calculated based on the coordinates of multiple key points in the posture model near the position corresponding to the end endpoint.

8. The reconstruction method according to claim 5, characterized in that: For the stretchable limb segment, the length of the virtual skeletal segment is axially scaled so that the magnitude of the third vector is equal to the magnitude of the fourth vector.

9. The reconstruction method according to claim 5, characterized in that: For the non-extensible limb segment, the length of the virtual skeleton segment is set so that the magnitude of the third vector is an average value of the magnitudes of the fourth vector in multiple frames.

10. The reconstruction method according to any one of claims 1 to 3, characterized in that: The first vector represents a vector from the end endpoint of the virtual skeleton segment at the tail to the front endpoint of the virtual skeleton segment at the head, and the second vector represents a vector from a key point at the tail to a key point at the head.

11. The reconstruction method according to any one of claims 1 to 3, characterized in that: The three-dimensional structural model is obtained by performing tomographic scanning on a sample of the processed static object using a three-dimensional imaging technology.

12. The reconstruction method according to any one of claims 1 to 3, characterized in that: The posture model is obtained by using a three-dimensional key point detection method to obtain the three-dimensional coordinates of the key points of each limb in all frames within the time period.

13. The reconstruction method according to any one of claims 1 to 3, characterized in that: The object includes any one of an animal, a movable facility, and a robot.

14. A system for reconstructing a three-dimensional dynamic model of an object, characterized in that: include: At least one processor is configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to any one of claims 1 to 13.

15. A device for reconstructing a three-dimensional dynamic model of an object, characterized in that: include: At least one processor is configured to execute the method for reconstructing a three-dimensional dynamic model of an object according to any one of claims 1 to 13. 16 . A storage medium storing a computer program for causing a computer to execute the method for reconstructing a three-dimensional dynamic model of an object according to claim 1 .

Images