A method and apparatus for constructing a three-dimensional model of a moving object

Abstract

The application discloses a kind of three-dimensional model construction method and equipment of moving object, comprising: the three-dimensional data information of target object is collected, and according to preset format coding, to obtain basic data set;Neural network model is constructed in advance, the neural network model includes sequentially arranged ordinary convolution layer, per-channel convolution layer, SE module layer, channel confusion operation and point-by-point convolution layer;The basic data set is input into the neural network model based on the, and based on the merging operation between the output of the point-by-point convolution layer and input data, to train the neural network model, after training, the required three-dimensional model is obtained.The method of the application solves the problem that the model constructed when capturing moving object is distorted and does not conform to common sense by using kinematics principle, estimating the rotation position of part of object, establishing related constraint loss function and using neural network for supervised learning.

Description

Technical Field

[0001] The present application relates to the field of data modeling technology, and in particular to a method and device for constructing a three-dimensional model of a moving object. Background Art

[0002] If depth cameras and sensors are used for motion capture, more specialized equipment is required, which is expensive and difficult to operate. The later rendering process is complicated and takes a long time to render. The relevant development process requires a large number of professionals to operate, and the rendered model can only be used in that specific environment and does not have environmental portability and universality. If deep learning is used, due to the lack of a large amount of relevant annotated data, accurate object target motion capture cannot be achieved after the model training is completed. In addition, due to the lack of real information about the movement and rotation of relevant nodes, some forms that are not in line with common sense will be generated. If you want to improve the learning and training effect of the entire model, you need to collect a large amount of 3D annotated data. However, collecting data requires a related 3D scanner, and it takes a lot of manpower to collect object motion data in various situations, which makes it more difficult to establish data and the development cost is too high.

[0003] Most existing 3D object motion modeling methods rely on real-time capture and rendering using depth cameras and sensors. However, this involves a wide variety of equipment, is expensive, and complex to operate. With the development of deep learning, related model construction has adopted deep learning training methods. However, deep learning requires a large amount of real-world data. Insufficient data leads to poor model fit, unrealistic model construction, and poor real-time model construction performance. Summary of the Invention

[0004] The embodiments of the present application provide a method and device for constructing a three-dimensional model of a moving object. By using the principles of kinematics and estimating the rotational position of the object part, a relevant constraint loss function is established, and a neural network is used for supervised learning, thereby solving the problem that the model constructed when capturing the moving object is distorted and inconsistent with common sense.

[0005] The present invention provides a method for constructing a three-dimensional model of a moving object, including:

[0006] Collect three-dimensional data information of the target object and encode it according to a preset format to obtain a basic data set;

[0007] Pre-building a neural network model, wherein the neural network model includes a normal convolution layer, a channel-by-channel convolution layer, a cross-channel attention mechanism (SE) module layer, a channel confusion operation, and a point-by-point convolution layer arranged in sequence;

[0008] The neural network model is input based on the basic data set, and a merging operation is performed between the output of the point-by-point convolution layer and the input data to train the neural network model, and the required three-dimensional model is obtained after the training is completed.

[0009] Optionally, encoding according to a preset format to obtain a basic data set includes:

[0010] The collected three-dimensional data information is encoded in a quaternary format. The encoding process satisfies:

[0011]

[0012] in Represents the location information of the root key point, ω represents the direction in which the part where the root node is located extends, Represents the location information of the other key points of the object, ω ref Represents the extension direction of the remaining key points.

[0013] Optionally, also include:

[0014] Combine the position information of the root keypoint with the direction in which the part where the root node is located extends; and

[0015] Combining the position information of the remaining key points of the object with the extension directions of the remaining key points;

[0016] Eliminate combinations that do not conform to common sense and obtain new combinations To expand the basic data set

[0017] Optionally, the SE module layer includes a first branch and a second branch;

[0018] The first branch does not perform any calculation, but directly merges the result of the second branch with the result of the first branch.

[0019] The second branch includes a global pooling layer, a fully connected layer, a fully connected layer and a Sigmoid activation function arranged in sequence to assign different weights to the features of different channels.

[0020] Optionally, the process of training the neural network model includes constructing the following loss function:

[0021] The difference distance between the quaternary data: L cos =1-(δ GT ×δ -1 );

[0022] Direct difference between quaternary data: L1 = || δ GT -δ|| 2 ;

[0023] Loss value between the kinematic function and the three-dimensional coordinate point: L2=||δ GT -FK(δ)|| 2 , where FK(·) represents the kinematic function;

[0024] Normalized constraint loss: L3 = |1-||δ|| 2 |;

[0025] The total constraint loss satisfies: L = L cos +L1+L2+L3;

[0026] The neural network model is trained based on a total constraint loss.

[0027] Optionally, the convolution kernel size of the common convolution layer is 3×3; the convolution kernel size of the channel-by-channel convolution layer is 3×3; and the convolution kernel size of the point-by-point convolution layer is 1×1.

[0028] An embodiment of the present application further provides a computer device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the aforementioned method for constructing a three-dimensional model of a moving object are implemented.

[0029] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for constructing a three-dimensional model of a moving object are implemented.

[0030] The embodiment of the present application uses the principles of kinematics to estimate the rotational position of the object part, establishes a related constraint loss function, and uses a neural network for supervised learning, thereby solving the problem that the model constructed when capturing the moving object is distorted and inconsistent with common sense.

[0031] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0033] Figure 1 This is a basic flow chart of the method of the embodiment of the present application;

[0034] Figure 2 An example of channel-by-channel convolution of a neural network designed for embodiments of the present application;

[0035] Figure 3 An example of the SE module layer of a neural network designed for an embodiment of the present application;

[0036] Figure 4 An example of channel confusion for a neural network designed for an embodiment of the present application;

[0037] Figure 5 An overall example of a neural network designed for an embodiment of the present application. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] The present application provides a method for constructing a three-dimensional model of a moving object. Figure 1 Shown, including:

[0040] In step S101, three-dimensional data information of a target object is collected and encoded according to a preset format to obtain a basic data set. Specifically, three-dimensional data information of a portion of a real target object can be collected and the data can be annotated. Then, in some embodiments, encoding according to the preset format to obtain a basic data set includes:

[0041] The collected three-dimensional data information is encoded in a quaternary format. The encoding process satisfies:

[0042]

[0043] in Represents the location information of the root key point, ω represents the direction in which the part where the root node is located extends, Represents the location information of the other key points of the object, ω ref Represents the extension direction of the remaining key points.

[0044] Since each part and the object are separated independently during data encoding, in this embodiment, each part and the whole can be separated and combined separately. In some embodiments, it also includes: combining the position information of the root key point with the direction of the part where the root node is located; and combining the position information of the remaining key points of the object with the direction of the extension of the remaining key points; excluding combinations that do not conform to relevant common sense, that is, ω is combined into Similarly, Make a combination. Make a random free combination if the conditions are met The new combination is the data generated by simulation to expand the basic data set.

[0045] Merge δ and δ′, normalize the merged data set, and convert it into a matrix form.

[0046] In step S102, a neural network model is pre-constructed, and the neural network model includes a normal convolution layer, a channel-by-channel convolution layer, a cross-channel attention mechanism (SE) module layer, a channel confusion operation, and a point-by-point convolution layer arranged in sequence. In some embodiments, the convolution kernel size of the normal convolution layer is 3×3; the convolution kernel size of the channel-by-channel convolution layer is 3×3; and the convolution kernel size of the point-by-point convolution layer is 1×1. Specifically, the embodiment of the present application designs and constructs a neural network block layer to train the model. The block consists of the following five parts.

[0047] The first layer consists of ordinary convolution calculations with a convolution kernel size of 3×3. This layer is mainly used for feature extraction, extracting the features of relevant joint point information through convolution.

[0048] The second layer consists of channel-by-channel convolution, whose convolution kernel size is 3×3. The process of channel-by-channel convolution is as follows: Figure 2 As shown in the figure, designing channel-by-channel convolution can effectively reduce the amount of computation, thereby increasing the training convergence speed of the model and improving the inference construction of the model.

[0049] In some embodiments, as Figure 3 As shown in Figure 2, the third layer is the SE module layer. The main function of this layer is to assign different weights to the features of different channels, so that they have different attention levels in subsequent feature extraction. The SE module layer includes a first branch and a second branch;

[0050] The first branch does not perform any calculation, but directly merges the result of the second branch with the result of the first branch.

[0051] The second branch consists of a global pooling layer, a fully connected layer, a fully connected layer, and a sigmoid activation function, which assign different weights to features in different channels. Specifically, the features of the second branch first pass through a global pooling layer for channel dimensionality reduction and compression, then pass through two fully connected layers in succession, and finally pass through a sigmoid activation function to calculate the weights for different channels.

[0052] like Figure 4As shown, the present application designs a fourth-layer channel confusion structure to eliminate the feature barrier phenomenon between channels. Through channel confusion, the features between channels can be communicated, which is beneficial to the next step of feature matching.

[0053] The fifth layer is a point-by-point convolution layer, which realizes feature fusion through 1×1 convolution, plays a role in dimensionality reduction, and reduces the amount of related calculations.

[0054] In step S103, the neural network model is input based on the basic data set, and a merging operation is performed between the output of the point-by-point convolution layer and the input data, such as Figure 5 As shown, modules are superimposed as needed based on relevant computing power and effects to train the neural network model, and the required three-dimensional model is obtained after the training is completed.

[0055] The embodiment of the present application uses the principles of kinematics to estimate the rotational position of the object part, establishes a related constraint loss function, and uses a neural network for supervised learning, thereby solving the problem that the model constructed when capturing the moving object is distorted and inconsistent with common sense.

[0056] In some embodiments, the process of training the neural network model includes constructing the following loss function:

[0057] The difference distance between the quaternary data: L cos =1-(δ GT ×δ -1 );

[0058] Direct difference between quaternary data: L1 = || δ GT -δ|| 2 ;

[0059] Loss value between the kinematic function and the three-dimensional coordinate point: L2=||δ GT -FK(δ)|| 2 , where FK(·) represents the kinematic function.

[0060] Normalized constraint loss: L3 = |1-||δ|| 2 |;

[0061] The total constraint loss satisfies: L = L cos +L1+L2+L3;

[0062] The neural network model is trained based on the total constraint loss, and a three-dimensional model based on object motion capture is obtained after the training is completed.

[0063] The three-dimensional model construction method of the embodiment of the present application reduces the collection and annotation process of real three-dimensional data, reduces a lot of manpower and material resources, and reduces the cost of data collection and preparation. The overall operation process of the three-dimensional model construction method of the embodiment of the present application is simple, does not require the operation of special equipment and relevant professionals, and does not require the subsequent rendering and construction of the model. The three-dimensional model construction method of the embodiment of the present application has strong real-time reasoning of the model, short construction time, and is not prone to the occurrence of situations that are inconsistent with common sense during the process of real-time action capture and construction, and has strong authenticity.

[0064] An embodiment of the present application further provides a computer device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the aforementioned method for constructing a three-dimensional model of a moving object are implemented.

[0065] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for constructing a three-dimensional model of a moving object are implemented.

[0066] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0067] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0069] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are protected by this application.

Claims

1. A method for constructing a three-dimensional model of a moving object, characterized in that: include: Collect three-dimensional data information of the target object and encode it according to a preset format to obtain a basic data set; Pre-building a neural network model, wherein the neural network model includes a normal convolution layer, a channel-by-channel convolution layer, a cross-channel attention mechanism SE module layer, a channel confusion operation, and a point-by-point convolution layer arranged in sequence; Inputting the neural network model based on the basic data set, and performing a merging operation between the output of the point-by-point convolution layer and the input data to train the neural network model, and obtaining the desired three-dimensional model after the training is completed; Encode in the preset format to obtain the basic data set including: The collected three-dimensional data information is encoded in a quaternary format. The encoding process satisfies: in Represents the location information of the root key point, ω represents the direction in which the part where the root node is located extends, Represents the location information of the other key points of the object, ω ref Represents the extension direction of the remaining key points; Also includes: Combine the position information of the root keypoint with the direction in which the part where the root node is located extends; and Combining the position information of the remaining key points of the object with the extension directions of the remaining key points; Eliminate combinations that do not conform to common sense and obtain new combinations To expand the basic data set, ω is combined into Similarly, Make a combination and make a random free combination if the conditions are met The new combination is the data generated by simulation to expand the basic data set.

2. The method for constructing a three-dimensional model of a moving object according to claim 1, wherein: The SE module layer includes a first branch and a second branch; The first branch does not perform any calculation, but directly merges the result of the second branch with the result of the first branch. The second branch includes a global pooling layer, a fully connected layer, a fully connected layer and a Sigmoid activation function arranged in sequence to assign different weights to the features of different channels.

3. The method for constructing a three-dimensional model of a moving object according to claim 2, wherein: The process of training the neural network model includes constructing the following loss function: The difference distance between the quaternary data: L cos =1-(δ GT ×δ -1 ); Direct difference between quaternary data: L1 = || δ GT -δ|| 2 ; Loss value between the kinematic function and the three-dimensional coordinate point: L2=||δ GT -FK(δ)|| 2 , where FK(·) represents the kinematic function; Normalized constraint loss: L3 = |1-‖‖δ‖| 2 ‖; The total constraint loss satisfies: L = L cos +L1+L2+L3; The neural network model is trained based on a total constraint loss.

4. The method for constructing a three-dimensional model of a moving object according to claim 1, wherein: The convolution kernel size of the common convolution layer is 3×3; the convolution kernel size of the channel-by-channel convolution layer is 3×3; and the convolution kernel size of the point-by-point convolution layer is 1×1.

5. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for constructing a three-dimensional model of a moving object according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for constructing a three-dimensional model of a moving object according to any one of claims 1 to 4.

Images