3D model construction methods, devices, electronic equipment and storage media

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for constructing a three-dimensional model, the method comprising: obtaining two-dimensional information of a target image containing a target object, the two-dimensional information containing two-dimensional features corresponding to a plurality of key points of the target object, then fusing the two-dimensional information with the two-dimensional features corresponding to each key point, obtaining a fused feature corresponding to each key point, then performing position regression on the fused features corresponding to each key point, obtaining a plurality of two-dimensional key point coordinates, then performing three-dimensional mapping on the plurality of two-dimensional key point coordinates, obtaining a plurality of three-dimensional key point coordinates of the target object, and finally constructing a three-dimensional model of the target object according to the plurality of three-dimensional key point coordinates of the target object. The present disclosure improves the temporal consistency of the three-dimensional model of the target object finally obtained by the above method, thereby solving the problem of temporal inconsistency in the three-dimensional reconstruction of the target object in the video.

Description

Technical Field

[0001] The present disclosure relates to three-dimensional reconstruction technology, and in particular to a three-dimensional model construction method, device, electronic device and storage medium. Background Art

[0002] 3D reconstruction is an important research direction in the field of computer vision, and 3D reconstruction based on a single frame image has very important theoretical significance and application value. 3D reconstruction can build 3D models of various target objects. For example, the target object can be a hand or a face. By obtaining the coordinates of multiple 3D key points of the target object, the 3D model of the target object can be constructed.

[0003] In the related art, three-dimensional reconstruction of the target object in the video can be completed by realizing three-dimensional reconstruction of the target object based on a single-frame image. However, when the three-dimensional reconstruction of the target object is realized through a single-frame image, usually only the reconstruction accuracy is considered, while the temporal consistency of the three-dimensional reconstruction of the target object is ignored, resulting in the problem of temporal inconsistency in the three-dimensional reconstruction of the target object in the video. Summary of the invention

[0004] The present disclosure provides a three-dimensional model construction method, device, electronic device and storage medium to at least solve the problem of inconsistent timing in the three-dimensional reconstruction of target objects in videos in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a three-dimensional model construction method is provided, comprising:

[0006] Acquire two-dimensional information of a target image containing a target object, wherein the two-dimensional information contains two-dimensional features corresponding to a plurality of key points of the target object;

[0007] The two-dimensional information is respectively fused with the two-dimensional features corresponding to each key point to obtain a fused feature corresponding to each key point;

[0008] Perform position regression on the fusion features corresponding to each key point to obtain multiple two-dimensional key point coordinates;

[0009] Performing three-dimensional mapping on the plurality of two-dimensional key point coordinates respectively to obtain the plurality of three-dimensional key point coordinates of the target object;

[0010] A three-dimensional model of the target object is constructed according to the coordinates of the multiple three-dimensional key points of the target object.

[0011] Optionally, the method further comprises:

[0012] Acquire multiple groups of sample target images containing the target object, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target object;

[0013] Input the sample target image into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object;

[0014] The preset model is trained according to the coordinates of a plurality of sample two-dimensional key points, sample three-dimensional key points and target three-dimensional key points of the target object to obtain a three-dimensional coordinate prediction model;

[0015] The step of obtaining two-dimensional information of a target image including a target object until coordinates of a plurality of three-dimensional key points are obtained includes:

[0016] The target image containing the target object is input into the three-dimensional coordinate prediction model to obtain a plurality of three-dimensional key point coordinates of the target object.

[0017] Optionally, the preset model is trained according to a plurality of sample two-dimensional key point coordinates, sample three-dimensional key point coordinates and target three-dimensional key point coordinates of the target object to obtain a three-dimensional coordinate prediction model, including:

[0018] Obtaining a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0019] Obtaining an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0020] With the goal of minimizing the consistency sample distance and minimizing the accuracy sample distance, the preset model is trained according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

[0021] Optionally, obtaining the consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates includes:

[0022] Performing data augmentation on the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional augmented sample;

[0023] Performing data augmentation on the coordinates of multiple sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional augmented sample;

[0024] Calculating the distances between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance;

[0025] Calculating the distances between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance;

[0026] The consistent sample distance is obtained according to the first sample distance and the second sample distance.

[0027] Optionally, performing three-dimensional mapping on the multiple two-dimensional key point coordinates to obtain the multiple three-dimensional key point coordinates of the target object includes:

[0028] According to the image features corresponding to the target image, posture pooling is performed on the coordinates of the multiple two-dimensional key points to obtain a set of two-dimensional key point feature vectors;

[0029] Obtaining multiple network vertex features of the target object according to the two-dimensional key point feature vector set and the current variable matrix;

[0030] Convolution processing is performed on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0031] Optionally, obtaining two-dimensional information of a target image containing a target object includes:

[0032] Cropping the target image containing the target object to obtain a target image block containing the target object;

[0033] Extracting features of a target image block containing the target object to obtain image features corresponding to the target image block;

[0034] The image features are upsampled to obtain two-dimensional information of a target image containing a target object.

[0035] Optionally, cropping the target image containing the target object to obtain a target image block containing the target object includes:

[0036] Performing target object detection on the target image to obtain a detection frame representing a position of the target object in the target image;

[0037] Expand the detection frame by a preset distance to obtain a cropping frame;

[0038] The target image is cropped according to the cropping frame to obtain a target image block containing the target object.

[0039] According to a second aspect of an embodiment of the present disclosure, a three-dimensional model construction device is provided, the device comprising:

[0040] A first acquisition module is configured to acquire two-dimensional information of a target image containing a target object, wherein the two-dimensional information includes two-dimensional features corresponding to a plurality of key points of the target object;

[0041] A fusion module is configured to fuse the two-dimensional information with the two-dimensional features corresponding to each key point to obtain a fusion feature corresponding to each key point;

[0042] The regression module is configured to perform position regression on the fused features corresponding to each key point to obtain multiple two-dimensional key point coordinates;

[0043] A mapping module is configured to perform three-dimensional mapping on the plurality of two-dimensional key point coordinates to obtain a plurality of three-dimensional key point coordinates of the target object;

[0044] The construction module is configured to construct a three-dimensional model of the target object according to the coordinates of multiple three-dimensional key points of the target object.

[0045] Optionally, the device further comprises:

[0046] A second acquisition module is configured to acquire multiple groups of sample target images containing the target object, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target object;

[0047] An input module is configured to input the sample target image into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object;

[0048] A training module is configured to train the preset model according to a plurality of sample two-dimensional key point coordinates, sample three-dimensional key point coordinates and target three-dimensional key point coordinates of the target object to obtain a three-dimensional coordinate prediction model;

[0049] The acquisition module is configured to input the target image containing the target object into the three-dimensional coordinate prediction model to obtain multiple three-dimensional key point coordinates of the target object.

[0050] Optionally, the training module includes:

[0051] A first obtaining submodule is configured to obtain a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0052] A second obtaining submodule is configured to obtain an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0053] The training submodule is configured to minimize the consistency sample distance and the accuracy sample distance as the goal, and train the preset model according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

[0054] Optionally, the first obtaining submodule includes:

[0055] A first amplification unit is configured to perform data amplification on coordinates of a plurality of sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional amplified sample;

[0056] A second amplification unit is configured to perform data amplification on coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional amplified sample;

[0057] A first obtaining unit is configured to calculate the distance between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance;

[0058] A second obtaining unit is configured to calculate the distance between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance;

[0059] The third obtaining unit is configured to obtain the consistent sample distance according to the first sample distance and the second sample distance.

[0060] Optionally, the mapping module includes:

[0061] A gesture pooling module is configured to perform gesture pooling on the plurality of two-dimensional key point coordinates according to image features corresponding to the target image to obtain a set of two-dimensional key point feature vectors;

[0062] A third obtaining submodule is configured to obtain a plurality of network vertex features of the target object according to the two-dimensional key point feature vector set and the current variable matrix;

[0063] The convolution submodule is configured to perform convolution processing on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0064] Optionally, the first acquisition module includes:

[0065] A cropping submodule is configured to crop the target image containing the target object to obtain a target image block containing the target object;

[0066] An extraction submodule is configured to perform feature extraction on a target image block containing the target object to obtain image features corresponding to the target image block;

[0067] The up-sampling submodule is configured to up-sample the image features to obtain two-dimensional information of the target image containing the target object.

[0068] Optionally, the cutting submodule includes:

[0069] a detection unit, configured to perform target object detection on the target image to obtain a detection frame representing a position of the target object in the target image;

[0070] An expansion unit is configured to expand the detection frame outward by a preset distance to obtain a cropping frame;

[0071] The cropping unit is configured to crop the target image according to the cropping frame to obtain a target image block containing the target object.

[0072] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0073] processor;

[0074] a memory for storing instructions executable by the processor;

[0075] The processor is configured to execute the instructions to implement the three-dimensional model construction method as described in the first aspect.

[0076] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the three-dimensional model construction method as described in the first aspect.

[0077] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer instructions, which, when executed by a processor, implement the three-dimensional model construction method as described in the first aspect.

[0078] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0079] The present invention obtains two-dimensional information of a target image containing a target object, wherein the two-dimensional information contains two-dimensional features corresponding to multiple key points of the target object, and then fuses the two-dimensional information with the two-dimensional features corresponding to each key point to obtain a fused feature corresponding to each key point, and then performs position regression on the fused features corresponding to each key point to obtain multiple two-dimensional key point coordinates, and then performs three-dimensional mapping on the multiple two-dimensional key point coordinates to obtain multiple three-dimensional key point coordinates of the target object, and finally constructs a three-dimensional model of the target object according to the multiple three-dimensional key point coordinates of the target object.

[0080] In the process of determining the coordinates of multiple two-dimensional key points of the target object according to the two-dimensional information of the target image, all the key points in the two-dimensional information are fused with the two-dimensional features corresponding to each key point, and the fused features of each key point obtained contain the two-dimensional features of other key points. The mutual relationship between multiple key points, that is, the position constraint, is taken into account, thereby considering the global factors. When the position regression is performed on the fused features corresponding to each key point, the effect of the global receptive field is achieved, and the temporal consistency of the obtained multiple two-dimensional key point coordinates is improved. In the whole process, high-level semantic information and underlying detailed image information are simultaneously considered, and the temporal consistency of the three-dimensional model of the target object finally obtained is improved, thereby solving the problem of temporal inconsistency in the three-dimensional reconstruction of the target object in the video.

[0081] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0083] Figure 1 is a flow chart of a three-dimensional model construction method according to an exemplary embodiment;

[0084] Figure 2 is a flow chart of a method for obtaining two-dimensional information according to an exemplary embodiment;

[0085] Figure 3 is a flow chart showing a method for cropping a target image block according to an exemplary embodiment;

[0086] Figure 4 is a flow chart of a three-dimensional mapping method according to an exemplary embodiment;

[0087] Figure 5 is a flow chart of a method for obtaining a three-dimensional coordinate prediction model according to an exemplary embodiment;

[0088] Figure 6 is a schematic diagram of the overall framework of a three-dimensional model construction method according to an exemplary embodiment;

[0089] Figure 7 is a block diagram of a three-dimensional model construction device according to an exemplary embodiment;

[0090] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0091] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0092] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0093] Figure 1 is a flowchart of a three-dimensional model construction method according to an exemplary embodiment. Figure 1 As shown, the following steps are included:

[0094] In step S11, two-dimensional information of a target image containing a target object is obtained, wherein the two-dimensional information contains two-dimensional features corresponding to a plurality of key points of the target object.

[0095] 3D reconstruction can construct 3D models of a variety of target objects. Hand 3D reconstruction (Hand Mesh Recovery) plays an important role in the AR / VR field, where the target object can be a hand, which can contain 778 3D key points. By determining the coordinates of the 3D key points of the hand and constructing the 3D model of the hand based on the triangular facets between these 3D key points, in the related technology, when performing 3D reconstruction of the movement of the user's hand, it is necessary to obtain the movement video of the user's hand in real time, and then perform 3D reconstruction of the hand for each frame of the movement video. However, when the 3D reconstruction of the target object is achieved through a single frame image, only the reconstruction accuracy is usually considered, while the temporal consistency of the target object after 3D reconstruction is ignored. That is, for In each frame of the image, the positions of multiple key points of each target object are interrelated. It is not a single individual and cannot appear at any position. There are certain constraints between the key points, and there are certain constraints between the key points of consecutive multi-frame images. The changes of the target object in the multi-frame image are coherent, but these relationships are ignored in the relevant technology, resulting in the problem of inconsistent timing in the three-dimensional reconstruction of the target object in the video, that is, the temporal consistency of the three-dimensional model of the target object is that the positional relationship between the multiple three-dimensional key points of the target object satisfies the mutual constraints between the key points of the target object and satisfies the temporal changes of the target object. Among them, the target image contains a complete target object, for example, a complete hand, and the existing heat map framework can be used to obtain the two-dimensional information corresponding to the target image, specifically, to obtain the two-dimensional features corresponding to the multiple key points of the target object.

[0096] In step S12, the two-dimensional information is fused with the two-dimensional features corresponding to each key point to obtain a fused feature corresponding to each key point.

[0097] In the above method, the target object corresponds to multiple key points, that is, the number and number of the key points of the target object are known. At this time, by performing feature extraction on each key point of the target image, the two-dimensional feature corresponding to each key point can be obtained, that is, the two-dimensional information contains the two-dimensional features corresponding to all key points. Then, the two-dimensional features corresponding to all key points are fused with the two-dimensional features corresponding to each key point respectively to obtain the fused feature corresponding to each key point. The fused feature contains the two-dimensional features corresponding to all key points, and also additionally contains the two-dimensional features corresponding to the corresponding key points. At this time, the fused feature corresponding to each key point is the same as the number of each key point, so as to obtain the two-dimensional key point coordinates of the corresponding key point.

[0098] In step S13, position regression is performed on the fusion features corresponding to each key point to obtain multiple two-dimensional key point coordinates.

[0099] Through the position regression method, the position regression is performed on the fused features corresponding to multiple key points. Each fused feature corresponds to each key point number one by one. Therefore, after the position regression of each fused feature, the two-dimensional key point coordinates of a key point corresponding to the key point number are obtained, and then the two-dimensional key point coordinates corresponding to all key points of the target object are obtained. Since each fused feature obtained contains the two-dimensional features of all key points, the global factors are taken into account. When the position regression is performed on multiple fused features, the effect of the global receptive field is achieved, and the temporal consistency of the obtained multiple two-dimensional key point coordinates is improved.

[0100] In step S14, three-dimensional mapping is performed on the multiple two-dimensional key point coordinates to obtain multiple three-dimensional key point coordinates of the target object.

[0101] After obtaining multiple two-dimensional key point coordinates, the multiple two-dimensional key point coordinates are three-dimensionally mapped. The specific three-dimensional mapping may include posture pooling, etc., so as to add the association relationship between each key point to each two-dimensional key point coordinate, and obtain the three-dimensional spatial features of each key point, so as to further obtain multiple three-dimensional key point coordinates of the target object.

[0102] In step S15, a three-dimensional model of the target object is constructed according to the coordinates of the multiple three-dimensional key points of the target object.

[0103] After obtaining the coordinates of multiple three-dimensional key points of the target object, the specific positions of the multiple three-dimensional key points can be obtained, and then, the multiple three-dimensional key points are connected through multiple triangular facets to construct a three-dimensional model of the target object.

[0104] By determining the coordinates of multiple two-dimensional key points of the target object according to the two-dimensional information of the target image, all the key points in the two-dimensional information are fused with the two-dimensional features corresponding to each key point, and the fused features of each key point obtained contain the two-dimensional features of other key points, thereby taking global factors into consideration. When performing position regression on the fused features corresponding to each key point, the effect of the global receptive field is achieved, and the temporal consistency of the obtained coordinates of multiple two-dimensional key points is improved. In the whole process, both high-level semantic information and underlying detailed image information are considered at the same time, and the temporal consistency of the three-dimensional model of the target object finally obtained is improved, thereby solving the problem of temporal inconsistency in the three-dimensional reconstruction of the target object in the video.

[0105] Figure 2 is a flow chart of a method for obtaining two-dimensional information according to an exemplary embodiment. Figure 2 As shown, based on the above technical solution, a specific method for obtaining two-dimensional information of a target image containing a target object may be:

[0106] In step S21, the target image containing the target object is cropped to obtain a target image block containing the target object;

[0107] In step S22, feature extraction is performed on the target image block containing the target object to obtain image features corresponding to the target image block;

[0108] In step S23, the image features are upsampled to obtain two-dimensional information of the target image containing the target object.

[0109] In this embodiment, before feature extraction is performed on the target image, the target image is first cropped to obtain a target image block containing the target object, so as to remove some extra image content in the target image, so as to reduce the data processing amount of feature extraction and avoid the influence of extra image content on the recognition of key points and feature extraction, thereby improving the accuracy of feature extraction.

[0110] Then, the target image block may be subjected to feature extraction through the first convolutional network. Specifically, the first convolutional network may be a feature extraction network in an existing heat map framework, so as to obtain image features corresponding to the target image block.

[0111] Finally, the image features can be upsampled through the second convolutional network to obtain two-dimensional information of the target image containing the target object, wherein the second convolutional network can also be a convolutional network in the heat map framework. If the target object is a hand, the convolutional network is a convolutional network for the hand, which is used to upsample the image features through the second convolutional network to obtain two-dimensional features corresponding to multiple key points of the hand. If the target object is a human face, two-dimensional features corresponding to multiple key points of the face are obtained, wherein the first convolutional network and the second convolutional network are jump-connected to ensure the accuracy of the coordinates of the multiple two-dimensional key points obtained subsequently.

[0112] Figure 3 is a flow chart of a method for cropping a target image block according to an exemplary embodiment. Figure 3 As shown, based on the above technical solution, the specific method of cropping the target image containing the target object to obtain the target image block containing the target object may be:

[0113] In step S31, target object detection is performed on the target image to obtain a detection frame representing the position of the target object in the target image.

[0114] The target image contains a complete target object. By performing target object detection on the target image, a detection frame representing the position of the target object in the target image can be obtained. Specifically, the traditional hand detection method can be used to determine the approximate position of the hand to obtain the detection frame.

[0115] In step S32, the detection frame is expanded outward by a preset distance to obtain a cropping frame.

[0116] With the detection frame as the center, the boundary of the detection frame is expanded outward by a preset distance. Specifically, the expansion distance can be determined according to the boundary length of the detection frame. For example, the preset distance can be 0.3 times the boundary length of the detection frame, thereby obtaining a cropping frame so that the target object can be completely located within the cropping frame.

[0117] In step S33, the target image is cropped according to the cropping frame to obtain a target image block containing the target object.

[0118] According to the cropping frame, the image within the cropping frame is cropped from the target image to obtain a target image block containing the target object.

[0119] Through the above method, the original image is cropped to obtain a target image block containing the target object, which can remove some extra image content in the target image to reduce the data processing amount of feature extraction and avoid the influence of extra image content on the recognition of key points and feature extraction, thereby improving the accuracy of feature extraction.

[0120] Figure 4 is a flow chart of a three-dimensional mapping method according to an exemplary embodiment. Figure 4 As shown, a method for performing three-dimensional mapping on multiple two-dimensional key point coordinates to obtain multiple three-dimensional key point coordinates of a target object may be:

[0121] In step S41, the plurality of two-dimensional key point coordinates are subjected to posture pooling according to image features corresponding to the target image to obtain a set of two-dimensional key point feature vectors.

[0122] The target image contains the posture features of the target object, that is, the image features corresponding to the target image contain the association information between each key point, which is a three-dimensional tensor. By performing posture pooling on each of the multiple two-dimensional key point coordinates, that is, taking the features at the two-dimensional position on the three-dimensional tensor from each two-dimensional key point coordinate, a two-dimensional key point feature vector is obtained for each two-dimensional key point, thereby obtaining a two-dimensional key point feature vector set containing multiple two-dimensional key point feature vectors.

[0123] In step S42, a plurality of network vertex features of the target object are obtained according to the two-dimensional key point feature vector set and the current variable matrix.

[0124] In the process of training the preset model, the variable matrix will be continuously adjusted to obtain a three-dimensional coordinate prediction model that meets accuracy and consistency. The current variable matrix is ​​the variable matrix corresponding to the three-dimensional coordinate prediction model after training. After the three-dimensional coordinate prediction model is trained, the variable matrix will no longer change. The set of two-dimensional key point feature vectors can be regarded as a matrix. By multiplying it with the current variable matrix, a matrix containing multiple network vertex features of the target object is obtained, so that multiple network vertex features of the target object can be split.

[0125] In step S43, convolution processing is performed on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0126] After obtaining multiple network vertex features, 3D decoding can be performed. The graph convolution method can be used to process multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0127] Figure 5 is a flow chart of a method for obtaining a three-dimensional coordinate prediction model according to an exemplary embodiment. Figure 5 As shown, based on the above technical solution, the above steps S11 to S14 are all to complete data processing through the trained three-dimensional coordinate prediction model. Therefore, before this, it is necessary to first obtain the trained three-dimensional coordinate prediction model, which specifically includes the following steps S51 to S53:

[0128] In step S51, a plurality of groups of sample target images containing the target object are obtained, wherein the sample target images carry the coordinates of a plurality of target three-dimensional key points of the target object;

[0129] In step S52, the sample target image is input into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object;

[0130] In step S53, the preset model is trained according to a plurality of sample two-dimensional key point coordinates, sample three-dimensional key point coordinates and target three-dimensional key point coordinates of the target object to obtain a three-dimensional coordinate prediction model.

[0131] In this embodiment, first, it is necessary to prepare multiple groups of sample target images containing target objects for training the preset model, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target objects, and the multiple target three-dimensional key point coordinates of the target objects carried by the sample target images are accurate key point coordinates. Then, a preset model is obtained, and the preset model is used to process the sample target images to obtain multiple sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target objects. Specifically, the preset model can process the sample target images containing the target objects through the methods in steps S11 to S14 in the above embodiments until multiple sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target objects are obtained. In the training process of each group, every time a sample target image containing a target object is input, the corresponding multiple sample two-dimensional key point coordinates and multiple sample three-dimensional key point coordinates can be obtained through the preset model. Then, according to the multiple sample two-dimensional key point coordinates of the target object, the sample three-dimensional key point coordinates and the target three-dimensional key point coordinates, the preset model is continued to be trained. Specifically, the parameters of the preset model are updated through the multiple sample two-dimensional key point coordinates, the sample three-dimensional key point coordinates and the target three-dimensional key point coordinates of the target object, so as to obtain a three-dimensional coordinate prediction model through multiple rounds of training.

[0132] Through the above method, existing samples carrying label marks are obtained to perform multiple rounds of training on the preset model. Specifically, multiple groups of sample target images containing target objects are used to train the preset model, and the sample target images carry multiple target three-dimensional key point coordinates of the target object, so that a three-dimensional coordinate prediction model can be trained. The three-dimensional coordinate prediction model takes the target image containing the target object as input, and can directly output multiple three-dimensional key point coordinates of the target object with high temporal consistency, thereby improving the data processing speed and quickly constructing a three-dimensional model of the target object. Since the processing method of the preset model adopts the method in steps S11 to S14 in the above embodiment, the temporal consistency of the constructed three-dimensional model of the target object can be improved.

[0133] Specifically, according to the coordinates of a plurality of sample two-dimensional key points of the target object, the coordinates of the sample three-dimensional key points and the coordinates of the target three-dimensional key points, the preset model is trained to obtain a specific method of the three-dimensional coordinate prediction model as follows:

[0134] Obtaining a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0135] Obtaining an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0136] With the goal of minimizing the consistency sample distance and minimizing the accuracy sample distance, the preset model is trained according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

[0137] In this embodiment, in the process of training the preset model, the output results can be supervised for consistency and accuracy, that is, the consistency sample distance corresponding to the sample target image is obtained according to the multiple sample two-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates corresponding to the sample target image, and the distance between the multiple target three-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates corresponding to the sample target image is calculated to obtain the accuracy sample distance corresponding to the sample target image, and according to the obtained consistency sample distance and accuracy sample distance, the parameters in the preset model are adjusted with the goal of minimizing the consistency sample distance and minimizing the accuracy sample distance, and the preset model is trained through multiple groups of sample target image blocks containing target objects until the preset conditions are met, and the training can be stopped to obtain a three-dimensional coordinate prediction model. The preset conditions can be that both the consistency sample distance and the accuracy sample distance converge, or the number of training times reaches a preset number, for example, the preset number of times is 40,000 times.

[0138] Among them, in order to improve the temporal consistency of the obtained multiple three-dimensional key point coordinates, a consistent sample distance can be obtained according to multiple sample two-dimensional key point coordinates and multiple sample three-dimensional key point coordinates, so that the parameters of the existing preset model can be updated through the consistent sample distance until the consistent sample distance is minimized, thereby improving the temporal consistency of the multiple three-dimensional key point coordinates output by the obtained three-dimensional coordinate prediction model.

[0139] In order to improve the accuracy of the obtained multiple three-dimensional key point coordinates, the distance between the target three-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates can be calculated according to the target three-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates to obtain the accuracy sample distance, so that the parameters of the existing preset model can be updated through the accuracy sample distance until the accuracy sample distance is minimized, thereby improving the accuracy of the multiple three-dimensional key point coordinates output by the obtained three-dimensional coordinate prediction model.

[0140] Specifically, according to the coordinates of the plurality of sample two-dimensional key points and the coordinates of the plurality of sample three-dimensional key points, a method for obtaining the consistent sample distance may be:

[0141] Performing data augmentation on the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional augmented sample;

[0142] Performing data augmentation on the coordinates of multiple sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional augmented sample;

[0143] Calculating the distances between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance;

[0144] Calculating the distances between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance;

[0145] The consistent sample distance is obtained according to the first sample distance and the second sample distance.

[0146] In the above method, data augmentation is performed on the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional augmented sample. The specific data augmentation method can be affine transformation or color jittering; data augmentation is performed on the coordinates of multiple sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional augmented sample. The augmentation methods that can be used are relative rotation and color jittering. By starting from a single sample, data augmentation is performed to simulate the temporal change scene of the target object, and an augmented sample for calculating consistency is obtained. Then, the distance between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image block and the two-dimensional augmented sample is calculated to obtain a first sample distance, and the distance between the coordinates of multiple sample three-dimensional key points corresponding to a single sample target image block and the three-dimensional augmented sample is calculated to obtain a second sample distance. Finally, according to the first sample distance and the second sample distance, a consistent sample distance is obtained, so that the temporal consistency of the output result of the preset model can be obtained, and the goal is to minimize the consistent sample distance during the training process, so as to improve the temporal consistency of the result output by the obtained three-dimensional coordinate prediction model.

[0147] In the above method, by starting from a single sample and performing data augmentation, the amount of sample used is reduced, and there is no need to find a large number of target images containing the target object, and the amount of annotation of the target image of the target object is reduced, which reduces the amount of data processing and saves time and human resources. In addition, data augmentation simulates the temporal change scenario of the target object, so that the coordinates of multiple three-dimensional key points output by the three-dimensional coordinate prediction model obtained after training can be more consistent with the temporal change law, thereby improving the temporal consistency of the constructed three-dimensional model of the target object.

[0148] Specifically, when data augmentation uses affine transformation and relative rotation, the corresponding calculation formulas for the first sample distance and the second sample distance may be:

[0149]

[0150]

[0151] in, is the first sample distance, is the second sample distance, are the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image block, is a two-dimensional amplification sample, V view1 V is the coordinates of multiple sample 3D key points corresponding to a single sample target image block, view2 For three-dimensional amplification samples, T 1→2 is an affine transformation, R 1→2 For relative rotation.

[0152] After obtaining the first sample distance and the second sample distance, a specific method for obtaining the consistent sample distance based on the first sample distance and the second sample distance may be: directly adding the first sample distance and the second sample distance to obtain the consistent sample distance; or setting weights for the first sample distance and the second sample distance in advance, and then adding the results of the first sample distance and the second sample distance after weight calculation to obtain the consistent sample distance.

[0153] After obtaining the three-dimensional coordinate prediction model, steps S11 to S14 are:

[0154] A target image containing a target object is input into the three-dimensional coordinate prediction model to obtain a plurality of three-dimensional key point coordinates of the target object.

[0155] The target image containing the target object is directly input into the three-dimensional coordinate prediction model, and the three-dimensional coordinate prediction model executes the data processing steps from step S11 to step S14 to obtain the coordinates of multiple three-dimensional key points of the target object.

[0156] Figure 6 is a schematic diagram of the overall framework of a three-dimensional model construction method according to an exemplary embodiment. Figure 6 As shown, a specific implementation of a three-dimensional model construction method is provided, which may specifically include 2D encoding, 2D-3D mapping and 3D decoding; Figure 6 The MapReg (registration network) in is used for 2D pose prediction, specifically for obtaining multiple 2D key point coordinates based on the 2D information of the target image. Figure 6 The key point features in are two-dimensional key point feature vectors, and the vertex features are network vertex features.

[0157] Specifically, the 2D encoding process is as follows: first obtain a target image containing the target object, such as Figure 6 The target image containing the hand is taken and the approximate position of the hand is determined using a traditional hand detection method to obtain a detection frame. Then, with the detection frame as the center, the boundary of the detection frame is expanded by a certain multiple, such as 0.3 times, to obtain a target image block containing the hand.

[0158] Then, taking the target image block as input, a convolutional network is used to obtain two-dimensional information of the target image containing the target object. The two-dimensional information includes two-dimensional features corresponding to multiple key points of the target object. Specifically, the convolutional network of the existing heat map can be used to obtain the two-dimensional information of the target image containing the target object.

[0159] 2D-3D mapping specifically includes 2D pose prediction, pose pooling and pose-vertex mapping.

[0160] Specifically, 2D pose prediction includes:

[0161] The two-dimensional information is respectively fused with the two-dimensional features corresponding to each key point to obtain the fused features corresponding to each key point;

[0162] Perform position regression on the fused features corresponding to each key point to obtain multiple two-dimensional key point coordinates.

[0163] Pose pooling includes: performing pose pooling on the coordinates of the multiple two-dimensional key points according to the image features corresponding to the target image to obtain a set of two-dimensional key point feature vectors. That is, by taking the features at the two-dimensional position of each two-dimensional key point coordinate on the three-dimensional tensor, a two-dimensional key point feature vector of each two-dimensional key point is obtained, thereby obtaining a set of two-dimensional key point feature vectors containing multiple two-dimensional key point feature vectors.

[0164] The posture-vertex mapping includes: obtaining multiple network vertex features of the target object according to the set of two-dimensional key point feature vectors and the current variable matrix. Multiple network vertex features can be obtained using a learnable matrix multiplication. Specifically, during the training of the preset model, the variable matrix will be continuously adjusted to obtain a three-dimensional coordinate prediction model that meets accuracy and consistency. The current variable matrix is ​​the variable matrix corresponding to the three-dimensional coordinate prediction model after the training is completed. After the three-dimensional coordinate prediction model is trained, the variable matrix no longer changes. The set of two-dimensional key point feature vectors can be regarded as a matrix. By multiplying it with the current variable matrix, a matrix containing multiple network vertex features of the target object is obtained, so that multiple network vertex features of the target object can be split.

[0165] The 3D decoding process is: convolution processing is performed on multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object. Specifically, after obtaining multiple network vertex features, 3D decoding can be performed. The graph convolution method can be used to process multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0166] Finally, a three-dimensional model of the target object can be constructed according to the obtained coordinates of multiple three-dimensional key points of the target object.

[0167] In the process of training the preset model to obtain the three-dimensional coordinate prediction model, accuracy supervision and consistency supervision can also be performed, specifically: according to the multiple sample two-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates corresponding to the sample target image, the consistency sample distance corresponding to the sample target image is obtained, and the distance between the multiple target three-dimensional key point coordinates and the multiple sample three-dimensional key point coordinates corresponding to the sample target image is calculated to obtain the accuracy sample distance corresponding to the sample target image, and according to the obtained consistency sample distance and accuracy sample distance, the parameters in the preset model are adjusted with the goal of minimizing the consistency sample distance and minimizing the accuracy sample distance.

[0168] Through the above method, before feature extraction is performed on the target image, the target image is first cropped to obtain a target image block containing the target object, so as to remove some extra image content in the target image, so as to reduce the data processing amount of feature extraction and avoid the influence of extra image content on the recognition of key points and feature extraction, thereby improving the accuracy of feature extraction.

[0169] In the process of determining the coordinates of multiple two-dimensional key points of the target object according to the two-dimensional information of the target image, all the key points in the two-dimensional information are fused with the two-dimensional features corresponding to each key point respectively, and the fused features of each key point obtained contain the two-dimensional features of other key points, thereby taking into account the global factors, that is, taking into account the natural logical relationship between multiple key points, and taking into account the position relationship factors between multiple key points when performing position regression on the fused features corresponding to each key point, achieving the effect of the global receptive field, and improving the temporal consistency of the obtained multiple two-dimensional key point coordinates. In the whole process, both high-level semantic information and underlying detailed image information are considered, and the temporal consistency of the three-dimensional model of the target object finally obtained is improved, thereby solving the problem of temporal inconsistency in the three-dimensional reconstruction of the target object in the video.

[0170] Moreover, during the model training process, by starting from a single sample and performing data augmentation, the amount of samples used is reduced, and there is no need to find a large number of target images containing the target object, and the amount of annotation of the target image of the target object is reduced, which reduces the amount of data processing and saves time and human resources. In addition, data augmentation simulates the temporal change scenario of the target object, so that the coordinates of multiple three-dimensional key points output by the three-dimensional coordinate prediction model obtained after training can be more consistent with the temporal change law, thereby improving the temporal consistency of the constructed three-dimensional model of the target object.

[0171] Figure 7 FIG. 1 is a block diagram of a three-dimensional model construction device according to an exemplary embodiment. Figure 7The device includes a first acquisition module 71, a fusion module 72, a regression module 73, a mapping module 74 and a construction module 75.

[0172] The first acquisition module 71 is configured to acquire two-dimensional information of a target image containing a target object, wherein the two-dimensional information includes two-dimensional features corresponding to a plurality of key points of the target object;

[0173] The fusion module 72 is configured to fuse the two-dimensional information with the two-dimensional features corresponding to each key point to obtain a fusion feature corresponding to each key point;

[0174] The regression module 73 is configured to perform position regression on the fusion features corresponding to each key point to obtain multiple two-dimensional key point coordinates;

[0175] The mapping module 74 is configured to perform three-dimensional mapping on the plurality of two-dimensional key point coordinates to obtain the plurality of three-dimensional key point coordinates of the target object;

[0176] The construction module 75 is configured to construct a three-dimensional model of the target object according to the coordinates of multiple three-dimensional key points of the target object.

[0177] Optionally, the device further comprises:

[0178] A second acquisition module is configured to acquire multiple groups of sample target images containing the target object, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target object;

[0179] An input module is configured to input the sample target image into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object;

[0180] A training module is configured to train the preset model according to a plurality of sample two-dimensional key point coordinates, sample three-dimensional key point coordinates and target three-dimensional key point coordinates of the target object to obtain a three-dimensional coordinate prediction model;

[0181] The acquisition module is configured to input the target image containing the target object into the three-dimensional coordinate prediction model to obtain multiple three-dimensional key point coordinates of the target object.

[0182] Optionally, the training module includes:

[0183] A first obtaining submodule is configured to obtain a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0184] A second obtaining submodule is configured to obtain an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates;

[0185] The training submodule is configured to minimize the consistency sample distance and the accuracy sample distance as the goal, and train the preset model according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

[0186] Optionally, the first obtaining submodule includes:

[0187] A first amplification unit is configured to perform data amplification on coordinates of a plurality of sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional amplified sample;

[0188] A second amplification unit is configured to perform data amplification on coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional amplified sample;

[0189] A first obtaining unit is configured to calculate the distance between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance;

[0190] A second obtaining unit is configured to calculate the distance between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance;

[0191] The third obtaining unit is configured to obtain the consistent sample distance according to the first sample distance and the second sample distance.

[0192] Optionally, the mapping module includes:

[0193] A gesture pooling module is configured to perform gesture pooling on the plurality of two-dimensional key point coordinates according to image features corresponding to the target image to obtain a set of two-dimensional key point feature vectors;

[0194] A third obtaining submodule is configured to obtain a plurality of network vertex features of the target object according to the two-dimensional key point feature vector set and the current variable matrix;

[0195] The convolution submodule is configured to perform convolution processing on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

[0196] Optionally, the first acquisition module includes:

[0197] A cropping submodule is configured to crop the target image containing the target object to obtain a target image block containing the target object;

[0198] An extraction submodule is configured to perform feature extraction on a target image block containing the target object to obtain image features corresponding to the target image block;

[0199] The upsampling submodule is configured to upsample the image features to obtain two-dimensional information of the target image containing the target object.

[0200] Optionally, the cutting submodule includes:

[0201] a detection unit, configured to perform target object detection on the target image to obtain a detection frame representing a position of the target object in the target image;

[0202] An expansion unit is configured to expand the detection frame outward by a preset distance to obtain a cropping frame;

[0203] The cropping unit is configured to crop the target image according to the cropping frame to obtain a target image block containing the target object.

[0204] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0205] Figure 8 5 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0206] Reference Figure 8 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0207] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0208] The memory 504 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0209] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0210] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0211] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0212] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0213] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0214] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 5G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0215] In an exemplary embodiment, the electronic device 500 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned three-dimensional model construction method.

[0216] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to complete the above-mentioned three-dimensional model construction method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0217] In an exemplary embodiment, a computer program product is also provided, the computer program product including computer instructions, the computer instructions can be executed by the processor 520 of the electronic device 500 to complete the above-mentioned three-dimensional model construction method. Optionally, the computer instructions can be stored in a storage medium of the electronic device 500, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0218] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0219] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A three-dimensional model construction method, characterized in that: The method comprises: Acquire two-dimensional information of a target image containing a target object, wherein the two-dimensional information contains two-dimensional features corresponding to a plurality of key points of the target object; The two-dimensional information is respectively fused with the two-dimensional features corresponding to each key point to obtain a fused feature corresponding to each key point; Perform position regression on the fusion features corresponding to each key point to obtain multiple two-dimensional key point coordinates; Performing three-dimensional mapping on the plurality of two-dimensional key point coordinates respectively to obtain the plurality of three-dimensional key point coordinates of the target object; Constructing a three-dimensional model of the target object according to the coordinates of the multiple three-dimensional key points of the target object; Acquire multiple groups of sample target images containing the target object, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target object; Input the sample target image into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object; The preset model is trained according to the coordinates of a plurality of sample two-dimensional key points, sample three-dimensional key points and target three-dimensional key points of the target object to obtain a three-dimensional coordinate prediction model; The step of obtaining two-dimensional information of a target image including a target object until coordinates of a plurality of three-dimensional key points are obtained includes: The target image containing the target object is input into the three-dimensional coordinate prediction model to obtain a plurality of three-dimensional key point coordinates of the target object.

2. The method according to claim 1, characterized in that The preset model is trained according to the coordinates of a plurality of sample two-dimensional key points of the target object, the coordinates of the sample three-dimensional key points and the coordinates of the target three-dimensional key points to obtain a three-dimensional coordinate prediction model, including: Obtaining a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates; Obtaining an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates; With the goal of minimizing the consistency sample distance and minimizing the accuracy sample distance, the preset model is trained according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

3. The method according to claim 2, characterized in that Obtaining a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates, including: Performing data augmentation on the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional augmented sample; Performing data augmentation on the coordinates of multiple sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional augmented sample; Calculating the distances between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance; Calculating the distances between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance; The consistent sample distance is obtained according to the first sample distance and the second sample distance.

4. The method according to claim 1, characterized in that Performing three-dimensional mapping on the multiple two-dimensional key point coordinates to obtain multiple three-dimensional key point coordinates of the target object includes: According to the image features corresponding to the target image, posture pooling is performed on the coordinates of the multiple two-dimensional key points to obtain a set of two-dimensional key point feature vectors; Obtaining multiple network vertex features of the target object according to the two-dimensional key point feature vector set and the current variable matrix; Convolution processing is performed on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

5. The method according to claim 1, characterized in that Acquire the two-dimensional information of the target image containing the target object, including: Cropping the target image containing the target object to obtain a target image block containing the target object; Extracting features of a target image block containing the target object to obtain image features corresponding to the target image block; The image features are upsampled to obtain two-dimensional information of a target image containing a target object.

6. The method according to claim 5, characterized in that Cropping the target image containing the target object to obtain a target image block containing the target object includes: Performing target object detection on the target image to obtain a detection frame representing a position of the target object in the target image; Expand the detection frame by a preset distance to obtain a cropping frame; The target image is cropped according to the cropping frame to obtain a target image block containing the target object.

7. A three-dimensional model construction device, characterized in that: The device comprises: A first acquisition module is configured to acquire two-dimensional information of a target image containing a target object, wherein the two-dimensional information includes two-dimensional features corresponding to a plurality of key points of the target object; A fusion module is configured to fuse the two-dimensional information with the two-dimensional features corresponding to each key point to obtain a fusion feature corresponding to each key point; The regression module is configured to perform position regression on the fused features corresponding to each key point to obtain multiple two-dimensional key point coordinates; A mapping module is configured to perform three-dimensional mapping on the plurality of two-dimensional key point coordinates to obtain a plurality of three-dimensional key point coordinates of the target object; A construction module, configured to construct a three-dimensional model of the target object according to the coordinates of a plurality of three-dimensional key points of the target object; A second acquisition module is configured to acquire multiple groups of sample target images containing the target object, wherein the sample target images carry multiple target three-dimensional key point coordinates of the target object; An input module is configured to input the sample target image into a preset model to obtain a plurality of sample two-dimensional key point coordinates and sample three-dimensional key point coordinates of the target object; A training module is configured to train the preset model according to a plurality of sample two-dimensional key point coordinates, sample three-dimensional key point coordinates and target three-dimensional key point coordinates of the target object to obtain a three-dimensional coordinate prediction model; The acquisition module is configured to input the target image containing the target object into the three-dimensional coordinate prediction model to obtain multiple three-dimensional key point coordinates of the target object.

8. The device according to claim 7, characterized in that The training module comprises: A first obtaining submodule is configured to obtain a consistent sample distance according to the plurality of sample two-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates; A second obtaining submodule is configured to obtain an accuracy sample distance according to the plurality of target three-dimensional key point coordinates and the plurality of sample three-dimensional key point coordinates; The training submodule is configured to minimize the consistency sample distance and the accuracy sample distance as the goal, and train the preset model according to the consistency sample distance and the accuracy sample distance to obtain the three-dimensional coordinate prediction model.

9. The device according to claim 8, characterized in that The first obtaining submodule includes: A first amplification unit is configured to perform data amplification on coordinates of a plurality of sample two-dimensional key points corresponding to a single sample target image to obtain a two-dimensional amplified sample; A second amplification unit is configured to perform data amplification on coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image to obtain a three-dimensional amplified sample; A first obtaining unit is configured to calculate the distance between the coordinates of multiple sample two-dimensional key points corresponding to a single sample target image and the two-dimensional augmented sample to obtain a first sample distance; A second obtaining unit is configured to calculate the distance between the coordinates of a plurality of sample three-dimensional key points corresponding to a single sample target image and the three-dimensional augmented sample to obtain a second sample distance; The third obtaining unit is configured to obtain the consistent sample distance according to the first sample distance and the second sample distance.

10. The device according to claim 7, characterized in that The mapping module comprises: A gesture pooling module is configured to perform gesture pooling on the plurality of two-dimensional key point coordinates according to image features corresponding to the target image to obtain a set of two-dimensional key point feature vectors; A third obtaining submodule is configured to obtain a plurality of network vertex features of the target object according to the two-dimensional key point feature vector set and the current variable matrix; The convolution submodule is configured to perform convolution processing on the multiple network vertex features to obtain multiple three-dimensional key point coordinates of the target object.

11. The device according to claim 7, characterized in that The first acquisition module includes: A cropping submodule is configured to crop the target image containing the target object to obtain a target image block containing the target object; An extraction submodule is configured to perform feature extraction on a target image block containing the target object to obtain image features corresponding to the target image block; The upsampling submodule is configured to upsample the image features to obtain two-dimensional information of the target image containing the target object.

12. The device according to claim 11, characterized in that The cutting submodule comprises: a detection unit, configured to perform target object detection on the target image to obtain a detection frame representing a position of the target object in the target image; An expansion unit is configured to expand the detection frame outward by a preset distance to obtain a cropping frame; The cropping unit is configured to crop the target image according to the cropping frame to obtain a target image block containing the target object.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional model construction method according to any one of claims 1 to 6.

14. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the three-dimensional model construction method as described in any one of claims 1 to 6.

15. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the three-dimensional model construction method according to any one of claims 1 to 6 is implemented.

Images