Model construction method, construction device, equipment, storage medium and program product

Abstract

The invention provides a three-dimensional human body model construction method and device, equipment, a storage medium and a program product, and relates to the technical field of three-dimensional modeling. The three-dimensional human body model construction method comprises the following steps: in response to an obtained monocular human body image, generating a normal graph of the monocular human body image; inputting the monocular human body image into a lightweight depth model to construct an initial cost body of the monocular human body image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost body to obtain enhanced depth information of the monocular human body image; a prior three-dimensional feature body of the human skeleton is generated based on the parameterized model corresponding to the monocular human body image; and constructing a three-dimensional human body model based on the normal graph, the enhanced depth information and the prior three-dimensional feature body of the human body skeleton. According to the technical scheme, the lightweight depth model is used, and the calculated amount and the number of parameters for constructing the three-dimensional human body model are reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional human body model construction method, a three-dimensional human body model construction device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Three-dimensional human body reconstruction based on a single perspective refers to the reconstruction of a three-dimensional human body model using images from a single perspective (such as photos or video frames). It is gradually and widely used in scenarios where service robots interact with humans and in augmented reality scenarios. The construction of a three-dimensional human body model requires the introduction of a large number of human-related features and a relatively cumbersome calculation process. While the calculation complexity is high, it will also cause memory occupation of the computing device. Therefore, there is an urgent need for a model construction solution that can reduce the calculation complexity and memory occupation while ensuring modeling accuracy.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a three-dimensional human body model construction method, a three-dimensional human body model construction device, an electronic device, a computer-readable storage medium and a computer program product, which at least to a certain extent overcome the problems of high modeling calculation complexity and memory occupation of computing devices in related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a method for constructing a three-dimensional human body model is provided, comprising: generating a normal map of a monocular human body image in response to an acquired monocular human body image; inputting the monocular human body image into a lightweight depth model to construct an initial cost volume of the monocular human body image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human body image; generating a priori three-dimensional feature volume of a human skeleton based on a parameterized model corresponding to the monocular human body image; and constructing the three-dimensional human body model based on the normal map, the enhanced depth information and the priori three-dimensional feature volume of the human skeleton.

[0007] In one embodiment of the present disclosure, the lightweight depth model includes a multi-scale feature extractor, a cost volume construction module and a lightweight cost volume regularization network module. The monocular human image is input into the lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model, and the initial cost volume is subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image, including: inputting the monocular human image and the reference image into the multi-scale feature extractor, performing discrete wavelet transform, so as to extract first image features of multiple depth levels of the monocular human image and second image features of multiple depth levels of the reference image; inputting the first image features and the second image features into the cost volume construction module, so as to apply affine transformation to the first image features at different viewing angles based on the second image features to construct the initial cost volume; and performing lightweight regularization processing on the initial cost volume based on the lightweight cost volume regularization network module to obtain the enhanced depth information.

[0008] In one embodiment of the present disclosure, the lightweight cost body regularization network module is constructed based on a three-dimensional depth-separable convolution operation, and the initial cost body is lightweight-regularized based on the lightweight cost body regularization network module to obtain the enhanced depth information, including: performing feature capture and fusion on the initial cost body based on a three-dimensional depth-separable convolution operation to obtain a corresponding intermediate cost body; performing information enhancement processing on the intermediate cost body in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the enhanced depth information.

[0009] In one embodiment of the present disclosure, the three-dimensional depth-separable convolution operation includes depth convolution and spatial point convolution, and the initial cost body is subjected to feature capture and fusion based on the three-dimensional depth-separable convolution operation to obtain a corresponding intermediate cost body, including: based on the depth convolution operation, inputting tensors of different depths in the initial cost body into corresponding channels for independent convolution to obtain corresponding independent channel features, wherein the independent channel features include depth features corresponding to human body pixels, local structural features of the human body, and global context features of the human body; based on the spatial point convolution operation, the independent channel features are subjected to feature integration in the depth direction and feature fusion in the local area to obtain an intermediate cost body of the initial cost body at the multiple depth levels.

[0010] In one embodiment of the present disclosure, information enhancement processing is performed on the intermediate cost body in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the enhanced depth information, including: configuring channel attention weights, spatial attention weights and depth attention weights for the intermediate cost body based on serial operations to obtain the enhanced depth information.

[0011] In one embodiment of the present disclosure, channel attention weights, spatial attention weights and depth attention weights are configured for the intermediate cost body based on serial operations to obtain the enhanced depth information, including: performing a nonlinear dimensionality reduction operation on the intermediate cost bodies of the multiple depth levels based on the channel attention mechanism to obtain the channel attention weights; configuring the channel attention weights for the intermediate cost body to obtain a first cost body; identifying matching points corresponding to the real human body structure in the first cost body based on the spatial attention mechanism to obtain the spatial attention weights; configuring the spatial attention weights for the first cost body to obtain a second cost body; performing a convolution operation along the depth dimension of the second cost body based on the depth attention mechanism to obtain the depth attention weights; configuring the depth attention weights for the second cost body to obtain the enhanced depth information.

[0012] In one embodiment of the present disclosure, a nonlinear dimensionality reduction operation is performed on the intermediate cost bodies of the multiple depth levels based on a channel attention mechanism to obtain channel attention weights, including: performing a maximum pooling operation on the intermediate cost body to obtain first local feature information; performing an average pooling operation on the intermediate cost body to obtain first overall feature information; respectively inputting the first local feature information and the first overall feature information into a multi-layer perceptron for aggregation to obtain aggregated features; and activating the aggregated features to obtain the channel attention weights.

[0013] In one embodiment of the present disclosure, matching points corresponding to real human body structures in the first cost body are identified based on the spatial attention mechanism to obtain the spatial attention weight, including: performing a maximum pooling operation on the first cost body to obtain second local feature information; performing an average pooling operation on the first cost body to obtain second overall feature information; filtering the second local feature information and the second overall feature information along the spatial direction to obtain key spatial position information related to deep matching; activating the key spatial position information to obtain the spatial attention weight.

[0014] In one embodiment of the present disclosure, a convolution operation is performed along the depth dimension of the second cost body based on a deep attention mechanism to obtain a depth attention weight, including: performing a maximum pooling operation on the second cost body to obtain third local feature information; performing an average pooling operation on the second cost body to obtain third overall feature information; performing depth enhancement on the third local feature information and the third overall feature information along the depth direction based on a specified convolution kernel, activating the depth enhanced information, and obtaining the depth attention weight.

[0015] In one embodiment of the present disclosure, a priori three-dimensional feature body of a human skeleton is generated based on a parameterized model corresponding to the monocular human image, including: obtaining three-dimensional spatial layout features of a human skeleton corresponding to the monocular human image based on the human skeleton structure in the parameterized model; obtaining two-dimensional projection coordinates of the three-dimensional spatial layout features; extracting corresponding two-dimensional skeleton features from the monocular human image based on the two-dimensional projection coordinates; splicing the two-dimensional skeleton features and the three-dimensional spatial layout features to obtain three-dimensional skeleton splicing features; and obtaining the priori three-dimensional feature body based on the obtained spatial query points and the three-dimensional skeleton splicing features, wherein the spatial query points are used to query relevant nodes in the three-dimensional skeleton splicing features.

[0016] In one embodiment of the present disclosure, the prior three-dimensional feature body is obtained based on the acquired spatial query point and the three-dimensional skeleton splicing feature, including: calculating the relative distance of the joint point in the three-dimensional skeleton splicing feature with respect to the spatial query point; detecting the correlation between the joint point and the spatial query point based on a threshold function; obtaining a feature expression of the human skeleton based on the spatial query point based on the detection result; and obtaining the prior three-dimensional feature body based on the feature expression.

[0017] In one embodiment of the present disclosure, the three-dimensional human body model is constructed based on the normal map, the enhanced depth information and the prior three-dimensional feature body of the human skeleton, including: the normal map of the monocular human body image includes a front normal map and a back normal map of the human body, the front normal map and the back normal map are used to provide front and back surface information of the human body, based on the projection points in the prior three-dimensional feature body, the front normal map, the back normal map and the monocular human body image are subjected to feature extraction to obtain two-dimensional features; based on the position points in the monocular human body image, the two-dimensional features are sampled to obtain sampling features; the sampling features, the enhanced depth information and the prior three-dimensional feature body are subjected to feature fusion of the projection points to obtain fusion features; the fusion features are used as additional parameters of the implicit function to solve the implicit function to construct the three-dimensional human body model.

[0018] According to another aspect of the present disclosure, a three-dimensional human body model construction device is provided, including: a first generation module, used to generate a normal map of the monocular human body image in response to an acquired monocular human body image; a lightweight depth processing module, used to input the monocular human body image into a lightweight depth model to construct an initial cost body of the monocular human body image based on the lightweight depth model, and perform lightweight regularization processing on the initial cost body to obtain enhanced depth information of the monocular human body image; a second generation module, used to generate a priori three-dimensional feature body of a human skeleton based on a parameterized model corresponding to the monocular human body image; and a construction module, used to construct the three-dimensional human body model based on the normal map, the enhanced depth information and the priori three-dimensional feature body of the human skeleton.

[0019] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the above-mentioned three-dimensional human body model construction method by executing the executable instructions.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned three-dimensional human body model construction method is implemented.

[0021] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned three-dimensional human body model construction method is implemented.

[0022] The three-dimensional human body model construction scheme provided by the embodiment of the present disclosure reduces the amount of calculation and the number of parameters by using a lightweight depth model, so that the normal execution of the construction of the three-dimensional human body model is guaranteed without occupying a large device memory. The regularization processing of the cost body can reduce the influence of noise and outliers, improve the quality of the cost body, and enable the obtained enhanced depth information to improve the accuracy of depth estimation. The normal map provides geometric information of the image surface and the prior knowledge of the human body structure provided by the prior three-dimensional feature body of the human skeleton. There is no need to store a large amount of intermediate data. While reducing the high computational complexity and memory usage in the construction of the three-dimensional human body model, the accuracy, robustness and computational efficiency of the constructed three-dimensional human body model are improved.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing a three-dimensional human body model construction system in an embodiment of the present disclosure;

[0026] Figure 2 A schematic flow chart showing a method for constructing a three-dimensional human body model in an embodiment of the present disclosure;

[0027] Figure 3 A schematic flow chart showing another method for constructing a three-dimensional human body model in an embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram showing encoding based on three-dimensional depth separable convolution in an embodiment of the present disclosure is shown;

[0029] Figure 5 A schematic flow chart showing another method for constructing a three-dimensional human body model in an embodiment of the present disclosure is shown;

[0030] Figure 6 A schematic diagram showing decoding based on a three-dimensional hybrid attention mechanism in an embodiment of the present disclosure is shown;

[0031] Figure 7 A schematic flow chart showing another method for constructing a three-dimensional human body model in an embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of another three-dimensional human body model construction device in an embodiment of the present disclosure is shown;

[0033] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] Three-dimensional human body reconstruction technology based on single perspective has been applied in some cases. For example, when a service robot interacts with a person, it needs to estimate the three-dimensional human body model of the interacting object from its own perspective; augmented reality applications need to estimate the three-dimensional model of the target human body from their own perspective.

[0037] From a visual perspective, since a single-view image can only provide visual information of the front of the human body, the three-dimensional reconstruction process is highly dependent on prior knowledge of human body geometry. In some embodiments, an RGB-D camera is a camera that can simultaneously acquire color (RGB) image information and depth information. It can match a statistically parameterized human body model with an RGB-D image to achieve reconstruction of a three-dimensional model of the human body. However, in images captured by an RGB-D camera, if part of the human body is blocked by an object, the visible surface area of ​​the human body is greatly reduced, which will lead to a reduction in the fitting accuracy of the model, and the image information of the blocked object may also introduce interference, resulting in a reduction in the three-dimensional reconstruction effect.

[0038] Based on the above defects, in the present disclosure, a lightweight deep model is introduced and cost volume regularization is added to the lightweight deep model to reduce the computational complexity of the model, thereby effectively reducing the number of model parameters and computing resource consumption.

[0039] To facilitate understanding, several terms involved in this application are first explained below.

[0040] Normal map: An image that represents the directional information of an object's surface. In three-dimensional space, each point on the object's surface has a normal vector, which is perpendicular to the surface at that point. The normal map usually uses color coding to represent the direction of the normal vector. For example, the three channels in the RGB color space can be used to represent the components of the normal vector on the three coordinate axes respectively.

[0041] Cost body: a data structure used to represent the matching cost under different assumptions. In computer vision, it is usually used to solve problems such as stereo matching and depth estimation. The cost body is usually a three-dimensional or higher-dimensional array, in which each element represents the matching cost between two images or features under a specific assumption (such as different depth values, disparity, etc.).

[0042] Regularization: It is a technique used to prevent overfitting and improve the generalization ability of the model during model training and optimization. Regularization is usually achieved by adding a penalty term to the objective function. This penalty term is related to the complexity of the model.

[0043] Affine transformation: A linear transformation that can translate, rotate, scale, shear, and perform other operations on points, vectors, and graphics in two-dimensional or three-dimensional space.

[0044] Figure 1 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application. The system includes: a plurality of tenant terminals 120 and a server terminal 140.

[0045] The tenant end 120 can be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the tenant end 120 can also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0046] Among them, the tenant terminal 120 may be installed with an application for providing three-dimensional human body model construction.

[0047] The tenant end 120 is connected to the server end 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0048] The server 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server 140 is used to provide background services for applications that provide three-dimensional human body model construction. Optionally, the server 140 undertakes the main computing work and the tenant 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work and the tenant 120 undertakes the main computing work; or, the tenant 120 and the server 140 adopt a distributed computing architecture for collaborative computing.

[0049] In some optional embodiments, the server 140 is used to store three-dimensional human body model construction program information.

[0050] Optionally, the logistics user terminals of the application programs installed in different tenant terminals 120 are the same, or the logistics user terminals of the application programs installed on the two tenant terminals 120 are logistics user terminals of the same type of application programs on different control system platforms. Based on the different terminal platforms, the specific forms of the logistics user terminals of the application programs may also be different, for example, the logistics user terminals of the application programs may be mobile logistics user terminals, PC logistics user terminals, or global wide area network (World Wide Web, Web) logistics user terminals, etc.

[0051] Those skilled in the art will appreciate that the number of the tenant terminals 120 may be more or less. For example, the number of the tenant terminals 120 may be only one, or the number of the tenant terminals may be dozens or hundreds, or more. The embodiment of the present application does not limit the number of terminals and the device type.

[0052] Optionally, the system may also include a management device ( Figure 1 (not shown), the management device is connected to the server 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0053] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0054] like Figure 2 As shown, a method for constructing a three-dimensional human body model according to an embodiment of the present disclosure includes:

[0055] Step S202: generating a normal map of the monocular human body image in response to the acquired monocular human body image.

[0056] In some embodiments, the normal map of a monocular human body image includes a front normal map and a back reverse map. The front normal map and the back normal map can describe the geometric shape of the human body. The normal map provides geometric shape information of the human body surface at different viewing angles. Each pixel in the normal map represents the normal direction of the corresponding surface point, that is, the vector perpendicular to the surface. By analyzing the normal map, the curvature, convexity and other geometric features of the human body surface can be inferred.

[0057] In some embodiments, human body surface details, such as skin texture, clothing wrinkles, etc., can also be determined based on the normal map.

[0058] In some embodiments, the front normal map and the back normal map may provide clues for depth estimation of a monocular human image, and may serve as constraints for constraining a depth estimation process of a monocular human image.

[0059] In some embodiments, the front normal map and the back normal map can be combined with other features (such as two-dimensional image features, skeleton structure prior information, etc.) to construct a three-dimensional human body model. By fusing multiple features, the performance of model construction can be improved.

[0060] Step S204: input the monocular human image into the lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model, and perform lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human image.

[0061] In some embodiments, a lightweight neural network architecture is used to build a lightweight deep model, such as building a lightweight deep model based on MobileNet or ShuffleNet.

[0062] In some embodiments, the cost volume is a three-dimensional data structure, which can be represented as D×H×W, where D represents the depth dimension, and H and W represent the height and width of the image, respectively.

[0063] In some embodiments, the initial cost volume is optimized through lightweight regularization processing, which can reduce the influence of noise and outliers to obtain more accurate depth information in the human body image.

[0064] Step S206, generating a priori three-dimensional feature volume of the human skeleton based on the parameterized model corresponding to the monocular human image.

[0065] In some embodiments, the parameterized model is used to provide shape and posture parameters of the human body, as well as corresponding body feature representations. The parameterized model may be SMPL (Skinned Multi-Person Linear Model) or the like.

[0066] In some embodiments, the parameterized model can also be used to determine additional normal prior information.

[0067] In some embodiments, the joint positions and connection relationships of the human body can be determined based on the human skeleton information in the parameterized model, and a three-dimensional feature body can be constructed based on this information, wherein each voxel can include feature information related to the human skeleton.

[0068] Step S208, constructing a three-dimensional human body model based on the normal map, the enhanced depth information and the prior three-dimensional feature volume of the human skeleton.

[0069] In some embodiments, implicit functions, mesh reconstruction, etc. can be used to construct a three-dimensional human body model using the integrated package information of the normal map, enhanced depth information, and a priori three-dimensional feature volume of the human skeleton.

[0070] In this embodiment, by using a lightweight depth model, the amount of calculation and the number of parameters are reduced, so that the normal execution of the construction of the three-dimensional human body model is guaranteed without occupying a large device memory. The regularization processing of the cost body can reduce the influence of noise and outliers, improve the quality of the cost body, and enable the obtained enhanced depth information to improve the accuracy of depth estimation. The normal map provides geometric information of the image surface and the prior knowledge of the human body structure provided by the prior three-dimensional feature body of the human skeleton. There is no need to store a large amount of intermediate data. While reducing the high computational complexity and memory usage in the construction of the three-dimensional human body model, the accuracy, robustness and computational efficiency of the constructed three-dimensional human body model are improved.

[0071] like Figure 3 As shown, in one embodiment of the present disclosure, the lightweight depth model includes a multi-scale feature extractor, a cost volume construction module, and a lightweight cost volume regularization network module. A monocular human image is input into the lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model, and the initial cost volume is subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image, including:

[0072] Step S302: input the monocular human image and the reference image into a multi-scale feature extractor to perform discrete wavelet transform to extract first image features of multiple depth levels of the monocular human image and second image features of multiple depth levels of the reference image.

[0073] Among them, the multi-scale feature extractor has an embedded discrete wavelet transform function. The source image and the reference image are sent to the multi-scale feature extractor to extract multi-level image features.

[0074] In some embodiments, a discrete wavelet transform is performed using a reference image as a comparison benchmark to decompose the image into sub-bands of different frequencies, thereby extracting features of different scales. By using the reference image as a benchmark, the differences and similarities between the monocular human body image and the reference image can be better captured.

[0075] Step S304: input the first image feature and the second image feature into a cost volume construction module, so as to apply affine transformation to the first image feature at different viewing angles based on the second image feature to construct an initial cost volume.

[0076] In some embodiments, the function of the cost volume construction module is to apply an affine transformation to the first image feature based on the second image feature at different viewing angles to construct an initial cost volume. By applying an affine transformation to the first image feature, the image changes at different viewing angles can be simulated to construct an initial cost volume.

[0077] Step S306 , performing lightweight regularization processing on the initial cost volume based on the lightweight cost volume regularization network module to obtain enhanced depth information.

[0078] In some embodiments, the lightweight cost body regularization network module can use some lightweight neural network structures, such as depth-separable convolution, bottleneck structure, etc. to optimize the initial cost body to obtain enhanced depth information to reduce the amount of calculation and the number of parameters.

[0079] In this embodiment, through a multi-scale feature extractor, image features can be extracted from different depth levels to better capture the details and structure of the image. In the cost body construction module, by applying an affine transformation to the first image feature, image changes under different perspectives can be simulated to construct a more accurate cost body. Furthermore, the lightweight cost body regularization network module adopts a lightweight neural network structure, which can reduce the amount of calculation and the number of parameters and improve computational efficiency.

[0080] In one embodiment of the present disclosure, a lightweight cost volume regularization network module is constructed based on a three-dimensional depth-separable convolution operation, and a lightweight regularization process is performed on the initial cost volume based on the lightweight cost volume regularization network module to obtain enhanced depth information, including:

[0081] Based on the three-dimensional depth-separable convolution operation, the features of the initial cost volume are captured and fused to obtain the corresponding intermediate cost volume.

[0082] Among them, three-dimensional depth-separable convolution refers to decomposing the three-dimensional convolution into convolution in the depth direction and convolution in the spatial direction, which is beneficial to reduce the amount of calculation and the number of parameters.

[0083] In some embodiments, the process of performing lightweight regularization on the initial cost body based on the lightweight cost body regularization network module to obtain enhanced depth information may include an encoding stage and a decoding stage, wherein the initial cost body is feature captured and fused based on a three-dimensional depth-separable convolution operation to obtain a corresponding intermediate cost body as the encoding stage.

[0084] In some embodiments, during the encoding process, when capturing features of the initial cost body, the convolution in the depth direction focuses on extracting features at different depth levels, while the convolution in the spatial direction captures the spatial structure information of the image, which can effectively extract the key features in the initial cost body and fuse them. After the three-dimensional depth-separable convolution operation, the corresponding intermediate cost body is obtained. The intermediate cost body includes information after preliminary feature extraction and fusion, which provides a basis for the subsequent decoding process.

[0085] Based on the three-dimensional hybrid attention mechanism, the intermediate cost body is enhanced in different dimensions to obtain enhanced depth information.

[0086] In some embodiments, in the decoding stage of the regularized network, in order to reduce the impact of feature matching on cost volume regularization and ensure its accuracy, a three-dimensional hybrid attention module can enhance important feature information and suppress the impact of abnormal data, remove noise and unnecessary data, and thus generate enhanced depth information.

[0087] In this embodiment, a lightweight cost body regularization network module is constructed using a three-dimensional deeply separable convolution operation. The deeply separable convolution greatly reduces the computational complexity, so that efficient depth information enhancement processing can be performed on resource-limited devices to obtain an intermediate cost body. While reducing the computational complexity, the accuracy of the regularization is guaranteed. Furthermore, the information enhancement processing of the intermediate cost body is performed in different dimensions through the three-dimensional hybrid attention mechanism, which can more effectively capture the key features of the depth information and enhance the robustness of the module.

[0088] In one embodiment of the present disclosure, a three-dimensional depth-separable convolution operation includes depth convolution and spatial point convolution. Based on the three-dimensional depth-separable convolution operation, feature capture and fusion are performed on the initial cost volume to obtain a corresponding intermediate cost volume, including:

[0089] Based on the deep convolution operation, the tensors of different depths in the initial cost volume are input into the corresponding channels for independent convolution to obtain the corresponding independent channel features. The independent channel features include the depth features corresponding to the human body pixels, the local structure features of the human body and the global context features of the human body.

[0090] In some embodiments, convolution is performed on the channel of each input tensor based on depth-wise convolution, and the convolution kernel of each channel is independent, without involving cross-channel convolution. That is, the convolution kernel is applied independently to each channel of the input tensor, which not only significantly reduces the number of parameters and computational complexity, but also allows the network to learn richer feature representations without increasing excessive computational burden.

[0091] In some embodiments, for an input tensor of size C in ×H×W(C in is the number of input channels, H and W are the height and width respectively), and when a deep convolution kernel of size K×K×K (K represents the size of the convolution kernel) is used for operation, the number of output channels is C out , the number of parameters is C in ×K×K×K, compared to the number of parameters C of the non-separable 3D convolution in ×K×K×K×C out , greatly reduced.

[0092] Based on the spatial point convolution operation, the independent channel features are integrated in the depth direction and the features of the local area are fused to obtain the intermediate cost volume of the initial cost volume at multiple depth levels.

[0093] In some embodiments, after the depth convolution, the point convolution combines the output of each channel. The point convolution effectively integrates the feature information in the depth direction by applying the convolution kernel in each local area. It can not only capture the local features in the spatial dimension, but also understand the distribution and changes of these features in the depth direction. It applies three-dimensional depth point-by-point convolution to the feature map independent of the channel to achieve the aggregation of channel information.

[0094] like Figure 4 As shown in the figure, the cost volume includes the dimension D in the depth direction, the dimension H in the height direction, and the dimension W in the width direction, and passes through a convolution kernel of size 3×3×1. That is, when performing depth-dimensional convolution, the convolution operation mainly extracts features with a 3×3 window in the height and width directions, and processes each slice one by one in the depth direction.

[0095] The convolution kernel of the spatial point convolution is 1×1×3, and it mainly extracts features around a spatial point with a window of 3 in the depth direction, while only operating on a single point in the height and width directions. This can extract and integrate feature information in the cost volume from different angles, providing more effective feature representation for subsequent tasks such as building a three-dimensional human body model.

[0096] In this embodiment, a complete convolution operation is formed by sequentially performing deep convolution and spatial point convolution. Deep convolution is used for feature decomposition operations, and spatial point convolution is used for feature recombination operations. The three-dimensional deep separable convolution not only reduces the number of parameters and computational complexity of the model, but also maintains or even improves the ability to extract features, and can help lightweight deep models learn nonlinear relationships between different channels. In the lightweight cost body regularization network module, three-dimensional deep separable convolution is used to extract and fuse cost body features from different perspectives, which is conducive to significantly reducing the number of parameters and computational complexity, so that the network can process large-scale three-dimensional data more efficiently. At the same time, due to its deep separation processing of features, the regularized network can better capture local details and global context information.

[0097] In one embodiment of the present disclosure, information enhancement processing is performed on the intermediate cost volume in different dimensions based on a three-dimensional hybrid attention mechanism to obtain enhanced depth information, including:

[0098] Based on serial operations, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body to obtain enhanced depth information.

[0099] Among them, serial operation refers to applying the weights generated in the previous step to the processing of the next step.

[0100] In some embodiments, the channel attention mechanism focuses on the importance of different channels of the intermediate cost body. Different channels may represent different feature information, such as human body appearance color, texture and edge shape. By calculating the channel attention weights, it is possible to automatically learn which channels are more critical for depth estimation, thereby performing targeted enhancement or suppression on different channels.

[0101] The spatial attention mechanism focuses on the importance of different spatial positions in the intermediate cost volume. Different spatial positions may have different depth information values. The spatial attention weights can highlight key spatial positions that may correspond to the real human body geometry, while suppressing less important spatial positions that may be affected by noise.

[0102] The deep attention mechanism focuses on the similarity confidence at different depth levels of the intermediate cost body. Different depth hypotheses correspond to different depth levels. The deep attention weights can adjust the credibility of different depth levels, enhance the depth levels that are more likely to appear, and suppress the depth levels that are less likely to appear.

[0103] In this embodiment, the intermediate cost volume is weighted in turn through the attention mechanisms in the three dimensions of channel, space and depth, so that the features that have important contributions to depth estimation can be highlighted more accurately. Different attention mechanisms focus on key information from different angles, thereby reducing the interference of irrelevant or noise information and improving the accuracy of depth information.

[0104] like Figure 5 As shown, in one embodiment of the present disclosure, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body based on serial operations to decode and obtain enhanced depth information, including:

[0105] Step S502: Perform a nonlinear dimensionality reduction operation on the intermediate cost bodies of multiple depth levels based on the channel attention mechanism to obtain channel attention weights.

[0106] In one embodiment of the present disclosure, a nonlinear dimensionality reduction operation is performed on intermediate cost bodies of multiple depth levels based on a channel attention mechanism to obtain channel attention weights, including: performing a maximum pooling operation on the intermediate cost body to obtain first local feature information; performing an average pooling operation on the intermediate cost body to obtain first overall feature information; respectively inputting the first local feature information and the first overall feature information into a multi-layer perceptron for aggregation to obtain aggregated features; and activating the aggregated features to obtain channel attention weights.

[0107] In some embodiments, the channel attention submodule is responsible for performing attention operations on the channel information in the cost body. First, a nonlinear dimensionality reduction operation is performed on the intermediate cost body, and the dimensionality reduction coefficient is set to 16 to obtain the channel attention weight W 1 ,The channel attention weight value is obtained by performing the maximum pooling and average pooling operations on the cost body, then aggregating it through the convolutional layer, and finally applying the activation function, as shown in formula (1).

[0108] W 1 =σ(M(MP(V))+M(AP(V))) (1)

[0109] Among them, W 1 represents the channel attention weight, σ is the activation function, M represents a multi-layer perceptron, MP represents maximum pooling, and AP represents average pooling.

[0110] Step S504: configure channel attention weights for the intermediate cost body to obtain a first cost body.

[0111] In some embodiments, the channel attention weight W 1 Multiply it with the intermediate cost volume V to get the first cost volume V after channel refinement 1 , as shown in formula (2).

[0112] V 1 =V×W 1 (2)

[0113] Step S506: Identify matching points in the first cost volume corresponding to the real human body structure based on the spatial attention mechanism, and obtain spatial attention weights.

[0114] In one embodiment of the present disclosure, matching points corresponding to real human body structures in a first cost body are identified based on a spatial attention mechanism to obtain spatial attention weights for enhancing occluded areas, including: performing a maximum pooling operation on the first cost body to obtain second local feature information; performing an average pooling operation on the first cost body to obtain second overall feature information; filtering the second local feature information and the second overall feature information along a spatial direction to enhance a depth representation of the occluded area in a monocular human body image to obtain key spatial position information related to depth matching; and activating the key spatial position information to obtain the spatial attention weight.

[0115] In some embodiments, the spatial attention submodule can effectively perceive the cost volume information at different spatial positions. By using the spatial attention mechanism, it can identify the matching points that are most likely to correspond to the real geometric structure, thereby improving the matching accuracy. A convolution kernel of size 1×7×7 is used to filter the first cost volume information along the spatial direction, while enhancing the feature representation and reducing noise interference, so that the convolution kernel can cover a larger spatial range, thereby capturing richer spatial context information, so that the key spatial positions related to deep matching can be more accurately identified and emphasized, while suppressing those less important or possibly caused by noise information, providing accurate spatial information for subsequent processing stages. The corresponding spatial attention weight is shown in formula (3).

[0116] W 2 =σ(conv 1×7×7 (MP(V 1 )+AP(V 1 ))) (3)

[0117] Among them, W 2 represents the spatial attention weight, σ is the activation function, conv 1×7×7 represents spatial convolution, MP represents maximum pooling, and AP represents average pooling.

[0118] Step S508: configure a spatial attention weight for the first cost volume to obtain a second cost volume.

[0119] Among them, calculate the second cost volume V 2 As shown in formula (4).

[0120] V 2 =V 1×W 2 (4)

[0121] Step S510: Perform a convolution operation along the depth dimension of the second cost volume based on the deep attention mechanism to obtain a deep attention weight.

[0122] In one embodiment of the present disclosure, a convolution operation is performed along the depth dimension of the second cost body based on the deep attention mechanism to obtain a depth attention weight, including: performing a maximum pooling operation on the second cost body to obtain third local feature information; performing an average pooling operation on the second cost body to obtain third overall feature information; performing depth enhancement on the third local feature information and the third overall feature information along the depth direction based on a specified convolution kernel, activating the depth enhanced information, and obtaining a depth attention weight.

[0123] In some embodiments, in order to improve the accuracy of matching information of different depth hypotheses at the same spatial position in 3D reconstruction, the deep domain attention module uses a 7×1×1 convolution kernel to operate along the depth dimension of the cost volume, thereby enhancing the depth information, enabling the model to more finely capture and distinguish features at different depths, effectively enhancing the network's ability to recognize and process depth information, thereby improving the accuracy of matching and the overall quality of reconstruction, and the corresponding deep attention weight is shown in formula (5). Finally, the three modules are cascaded to form a 3D hybrid attention module.

[0124] W 3 =σ(conv 7×1×1 (MP(V 2 )+AP(V 2 ))) (5)

[0125] Step S512: configure a depth attention weight for the second cost volume to obtain enhanced depth information.

[0126] Among them, the enhanced depth information V is calculated 3 As shown in formula (6).

[0127] V 3 =V 2 ×W 3 (6)

[0128] like Figure 6 As shown in the figure, the channel attention submodule is responsible for paying attention to the channel information in the cost body, the spatial attention submodule can effectively perceive the cost body information at different spatial positions, and the depth domain attention module operates along the depth dimension of the cost body by using convolution kernels, thereby enhancing the depth information.

[0129] In this embodiment, by integrating the effects of channel attention, spatial attention and depth attention, the model's attention to information of different dimensions in the cost body can be comprehensively enhanced. By integrating the attention weights of these three dimensions, the model can generate a final cost body tensor that integrates multiple aspects of information, that is, enhances depth information, and at the same time can suppress the impact of abnormal data on depth information, which is beneficial to optimize the regularization process of the cost body to improve the accuracy of three-dimensional human feature matching.

[0130] In one embodiment of the present disclosure, generating a priori three-dimensional feature volume of a human skeleton based on a parameterized model corresponding to a monocular human image includes:

[0131] Based on the human skeleton structure in the parameterized model, the three-dimensional spatial layout features of the human skeleton corresponding to the monocular human image are obtained; the two-dimensional projection coordinates of the three-dimensional spatial layout features are obtained; based on the two-dimensional projection coordinates, the corresponding two-dimensional skeleton features are extracted from the monocular human image; the two-dimensional skeleton features and the three-dimensional spatial layout features are spliced ​​to obtain the three-dimensional skeleton splicing features; based on the obtained spatial query points and the three-dimensional skeleton splicing features, a priori three-dimensional feature body is obtained, and the spatial query points are used to query related nodes in the three-dimensional skeleton splicing features.

[0132] In some embodiments, information related to the human body is input into a parametric model. The monocular human body image includes the appearance features, posture information, etc. of the human body. The parametric model generates three-dimensional spatial layout features of the human skeleton based on the input data, including the position of each joint, the length and direction of the bones, etc. The shape and structure of the human skeleton in three-dimensional space can be determined based on the three-dimensional spatial layout features.

[0133] Furthermore, based on the generated three-dimensional spatial layout information, the projection coordinates of each skeleton point on the two-dimensional plane are calculated, and the calculated two-dimensional projection coordinates are used to extract features corresponding to the skeleton points from the corresponding monocular human body image, and the extracted two-dimensional features are spliced ​​with the original three-dimensional features. Splicing can be achieved through a simple connection operation, combining the two-dimensional features and the three-dimensional features in a certain order to form a richer feature vector, and the spliced ​​feature vector is sent to the multi-layer perceptron, which determines whether a specific sampling point is located inside or outside the human body based on the input feature vector.

[0134] In this embodiment, a three-dimensional feature map is generated based on a parameterized model to obtain the three-dimensional spatial layout information of the skeleton. Next, the two-dimensional projection coordinates of these three-dimensional information are used to extract corresponding features from the corresponding two-dimensional feature map. In order to enhance the richness and accuracy of the features, the extracted two-dimensional features are spliced ​​with the original three-dimensional features. These spliced ​​features are sent to a multi-layer perceptron to determine whether a specific sampling point is located inside or outside the human body, thereby effectively improving the recognition ability of the human skeleton features.

[0135] In one embodiment of the present disclosure, obtaining a priori three-dimensional feature body based on the acquired spatial query point and the three-dimensional skeleton splicing feature includes:

[0136] Calculate the relative distance between the joint points in the three-dimensional skeleton splicing features and the spatial query points; detect the correlation between the joint points and the spatial query points based on the threshold function; obtain the feature expression of the human skeleton based on the spatial query points based on the detection results; and obtain the prior three-dimensional feature body based on the feature expression.

[0137] In some embodiments, the spatial query point can be an arbitrarily selected point in three-dimensional space. Its purpose is to calculate the relative position with the joint point, so as to extract features about the human skeleton. For each joint point and spatial query point, the relative distance between them is calculated, which can be calculated using the Euclidean distance formula.

[0138] In some embodiments, the feature expression of the human skeleton based on the spatial query point is shown in equations (7) to (9).

[0139] r k =[r k,1 ,……,r k,m ] (7)

[0140] r k，j =G(‖p k -J j ‖2∈R), j∈(1,m) (8)

[0141] Among them, r k represents the kth spatial query point p k Relative to the feature expression of each joint point, m refers to the number of joint points, k refers to the number of spatial query points, G(‖p k -J j ‖2∈R) represents the threshold function. By calculating the relative position of the control query point and each joint point, the feature expression of the spatial query point relative to each joint point can be obtained. By introducing the threshold function G(‖p k -J j ‖2∈R), which can reduce the influence of irrelevant nodes in feature expression.

[0142] The skeleton-based structural prior information is shown in formula (9):

[0143] H J (p k )=r k ,k=1,...,N+K (9)

[0144] Among them, N+K is the spatial query point obtained by the skeleton-based sampling strategy.

[0145] In one embodiment of the present disclosure, a 3D human body model is constructed based on a normal map, enhanced depth information, and a priori 3D feature volume of a human skeleton, including:

[0146] The normal map of a monocular human body image includes a front normal map and a back normal map of the human body. The front normal map and the back normal map are used to provide information on the front and back surfaces of the human body. Feature extraction is performed on the front normal map, the back normal map and the monocular human body image based on the projection points in the prior three-dimensional feature body to obtain two-dimensional features. Based on the position points in the monocular human body image, the two-dimensional features are sampled to obtain sampled features. The sampled features, enhanced depth information and features of the projection points of the prior three-dimensional feature body are fused to obtain fused features. The fused features are used as additional parameters of the implicit function to solve the implicit function to construct a three-dimensional human body model.

[0147] In some embodiments, the parameters of the parametric model can be used to generate a surface mesh of the human body. The surface mesh can be regarded as a three-dimensional object composed of many small triangles, which can accurately represent the surface shape of the human body. For the generated surface mesh, the normal vectors of each triangular facet on the front and back sides are calculated respectively. The normal vector is a vector perpendicular to the triangular facet, which represents the direction of the surface at that point. In the case of occlusion, a monocular human body image may not be able to fully display all parts of the human body. The front normal map and the back normal map can provide geometric information of the human body surface in different directions, so as to better understand the shape and position of the occluded part when the three-dimensional human body model is established.

[0148] In some embodiments, the acquired normal images of the front and back surfaces of the human body and the structural prior information of the skeleton are input into the model together with the two-dimensional image features. The model extracts key features that can accurately reflect the human body morphology through effective information fusion. Finally, these features are decoded using implicit functions to generate a detailed three-dimensional human body model.

[0149] The implicit function of pixel alignment in the human body 3D reconstruction method based on geometric prior information is shown in formula (10).

[0150]

[0151] Among them, X can be understood as a point or a position information in the three-dimensional human body model, x = π (X) is the projection of X on the two-dimensional image plane, and F L , B L are the front normal map and the back normal map respectively, I is the input two-dimensional monocular human image, Φ is the sampling function, which is used to obtain the feature corresponding to x on the obtained feature map according to the projection point x, H j (p k ) is the structural prior information based on the skeleton, Z is the depth value in the coordinate space, and g is the convolutional neural network used to extract features to obtain two-dimensional features.

[0152] Furthermore, the sampling features, enhanced depth information and prior three-dimensional feature bodies are fused at the projection point to obtain the fused features, which are then combined with the implicit function reconstruction module to complete the complete reconstruction of the three-dimensional human body. While making full use of the prior knowledge of the front and back normal maps and the skeleton, the accurate conversion from two-dimensional images to three-dimensional models is achieved through feature aggregation and implicit function decoding. By utilizing the constraints provided by the multi-attribute priors, the reconstruction process of the three-dimensional human body can be effectively standardized, thereby significantly improving the detail expression capability of the three-dimensional human body model.

[0153] like Figure 7 As shown, according to a three-dimensional human body model construction scheme of another embodiment of the present disclosure, a monocular human body image is respectively input into a normal map generation module 702, a lightweight depth model 704 and a parameterized model estimation module 706, the normal map generation module 702 outputs a normal map, the parameterized model estimation module 706 outputs a priori three-dimensional feature body of the human skeleton, the lightweight depth model 704 includes a multi-scale wavelet convolution feature extraction module 7042, a cost body construction module 7044 and a lightweight cost body regularization module 7046, and outputs enhanced depth information, the normal map, the enhanced depth information and the priori three-dimensional feature body of the human skeleton are input into a feature aggregation module 708, and aggregated features are obtained as additional parameters of the implicit function 710, thereby realizing accurate conversion from a two-dimensional image to a three-dimensional model.

[0154] In this embodiment, the input two-dimensional image is encoded through a lightweight deep model to obtain enhanced depth information, and the predicted parameterized model is converted into a prior three-dimensional feature body of the skeleton structure through a three-dimensional convolutional neural network. In addition, the front and back normal maps obtained by the normal map generation module, the front and back normal maps and their prior three-dimensional feature bodies of the skeleton structure and the two-dimensional image features including the enhanced depth information are used as input together, so as to be further integrated through feature aggregation technology, and finally the implicit function reconstruction module completes the complete reconstruction of the three-dimensional human body. By utilizing the constraints provided by the multi-attribute prior, the reconstruction process of the three-dimensional human body is effectively standardized, and the detail expression capability of the three-dimensional human body model is significantly improved.

[0155] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0156] Refer to the following Figure 8 A three-dimensional human body model construction device 800 according to an embodiment of the present disclosure is described. Figure 8 The three-dimensional human body model construction device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0157] The three-dimensional human body model construction device 800 is expressed in the form of a hardware module or a software module. The components of the three-dimensional human body model construction device 800 may include but are not limited to: a first generation module 802, which is used to generate a normal map of a monocular human body image in response to the acquired monocular human body image; a lightweight depth processing module 804, which is used to input the monocular human body image into a lightweight depth model to construct an initial cost body of the monocular human body image based on the lightweight depth model, and perform lightweight regularization on the initial cost body to obtain enhanced depth information of the monocular human body image; a second generation module 806, which is used to generate a priori three-dimensional feature body of a human skeleton based on a parameterized model corresponding to the monocular human body image; a construction module 808, which is used to construct a three-dimensional human body model based on the normal map, enhanced depth information and the priori three-dimensional feature body of the human skeleton.

[0158] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0159] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 2 The scheme described in .

[0160] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0161] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0162] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0163] The electronic device 900 may also communicate with one or more external devices 790 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0164] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0165] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to enable the electronic device to perform the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0166] According to the program product for implementing the above method according to the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0167] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0168] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0169] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0170] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the IoT terminal, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0171] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0172] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0173] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0174] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for constructing a three-dimensional human body model, characterized in that: include: In response to the acquired monocular human body image, generating a normal map of the monocular human body image; Inputting the monocular human image into a lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human image; Generate a priori three-dimensional feature volume of the human skeleton based on the parameterized model corresponding to the monocular human image; The three-dimensional human body model is constructed based on the normal map, the enhanced depth information and the prior three-dimensional feature volume of the human skeleton.

2. The method for constructing a three-dimensional human body model according to claim 1, characterized in that: The lightweight depth model includes a multi-scale feature extractor, a cost volume construction module and a lightweight cost volume regularization network module. The monocular human image is input into the lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model, and the initial cost volume is subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image, including: Inputting the monocular human body image and the reference image into the multi-scale feature extractor, and performing discrete wavelet transform to extract first image features of multiple depth levels of the monocular human body image and second image features of multiple depth levels of the reference image; Inputting the first image feature and the second image feature into the cost volume construction module, so as to apply affine transformation to the first image feature at different viewing angles based on the second image feature, so as to construct the initial cost volume; Based on the lightweight cost volume regularization network module, lightweight regularization processing is performed on the initial cost volume to obtain the enhanced depth information.

3. The method for constructing a three-dimensional human body model according to claim 2, characterized in that: The lightweight cost volume regularization network module is constructed based on a three-dimensional depth-separable convolution operation, and the initial cost volume is subjected to lightweight regularization processing based on the lightweight cost volume regularization network module to obtain the enhanced depth information, including: Based on a three-dimensional depth-separable convolution operation, feature capture and fusion are performed on the initial cost volume to obtain a corresponding intermediate cost volume; Based on the three-dimensional hybrid attention mechanism, information enhancement processing is performed on the intermediate cost volume in different dimensions to obtain the enhanced depth information.

4. The method for constructing a three-dimensional human body model according to claim 3, characterized in that: The three-dimensional depth-separable convolution operation includes depth convolution and spatial point convolution. Based on the three-dimensional depth-separable convolution operation, the initial cost volume is subjected to feature capture and fusion to obtain a corresponding intermediate cost volume, including: Based on the depth convolution operation, the tensors of different depths in the initial cost volume are input into the corresponding channels for independent convolution to obtain corresponding independent channel features, wherein the independent channel features include depth features corresponding to human body pixels, local structure features of the human body, and global context features of the human body; Based on the spatial point convolution operation, the independent channel features are respectively integrated in the depth direction and fused in the local area to obtain the intermediate cost volume of the initial cost volume at the multiple depth levels.

5. The method for constructing a three-dimensional human body model according to claim 3, characterized in that: Based on the three-dimensional hybrid attention mechanism, the intermediate cost volume is subjected to information enhancement processing in different dimensions to obtain the enhanced depth information, including: Based on a serial operation, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body to obtain the enhanced depth information.

6. The method for constructing a three-dimensional human body model according to claim 5, characterized in that: Configuring a channel attention weight, a spatial attention weight, and a depth attention weight for the intermediate cost volume based on a serial operation to obtain the enhanced depth information, including: Performing a nonlinear dimensionality reduction operation on the intermediate cost bodies of the multiple depth levels based on a channel attention mechanism to obtain the channel attention weights; Configuring the channel attention weight for the intermediate cost body to obtain a first cost body; Based on the spatial attention mechanism, identifying the matching points in the first cost volume corresponding to the real human body structure, obtaining the spatial attention weight; Configuring the spatial attention weight for the first cost volume to obtain a second cost volume; Performing a convolution operation along the depth dimension of the second cost volume based on a deep attention mechanism to obtain a deep attention weight; The depth attention weight is configured for the second cost volume to obtain the enhanced depth information.

7. The method for constructing a three-dimensional human body model according to claim 6, characterized in that: Performing a nonlinear dimensionality reduction operation on the intermediate cost bodies of the multiple depth levels based on the channel attention mechanism to obtain the channel attention weights includes: Performing a maximum pooling operation on the intermediate cost volume to obtain first local feature information; Performing an average pooling operation on the intermediate cost volume to obtain first overall feature information; Inputting the first local feature information and the first overall feature information into a multi-layer perceptron for aggregation respectively to obtain aggregated features; The aggregated features are activated to obtain the channel attention weight.

8. The method for constructing a three-dimensional human body model according to claim 6, characterized in that: Identifying matching points corresponding to the real human body structure in the first cost volume based on the spatial attention mechanism, and obtaining the spatial attention weight, including: Performing a maximum pooling operation on the first cost volume to obtain second local feature information; Performing an average pooling operation on the first cost volume to obtain second overall feature information; Filtering the second local feature information and the second overall feature information along a spatial direction to obtain key spatial position information related to depth matching; The key spatial position information is activated to obtain the spatial attention weight.

9. The method for constructing a three-dimensional human body model according to claim 6, characterized in that: Based on the deep attention mechanism, a convolution operation is performed along the depth dimension of the second cost volume to obtain a deep attention weight, including: Performing a maximum pooling operation on the second cost volume to obtain third local feature information; Performing an average pooling operation on the second cost volume to obtain third overall feature information; Based on the specified convolution kernel, the third local feature information and the third overall feature information are deeply enhanced along the depth direction, the depth enhanced information is activated, and the depth attention weight is obtained.

10. The method for constructing a three-dimensional human body model according to claim 2, characterized in that: Generating a priori three-dimensional feature volume of a human skeleton based on the parameterized model corresponding to the monocular human image includes: Obtaining a three-dimensional spatial layout feature of a human skeleton corresponding to the monocular human image based on the human skeleton structure in the parameterized model; Obtaining the two-dimensional projection coordinates of the three-dimensional spatial layout features; Extracting corresponding two-dimensional skeleton features from the monocular human body image based on the two-dimensional projection coordinates; Splicing the two-dimensional skeleton feature and the three-dimensional spatial layout feature to obtain a three-dimensional skeleton splicing feature; The priori three-dimensional feature volume is obtained based on the acquired spatial query point and the three-dimensional skeleton splicing feature, and the spatial query point is used to query the relevant nodes in the three-dimensional skeleton splicing feature.

11. The method for constructing a three-dimensional human body model according to claim 10, characterized in that: The prior three-dimensional feature body is obtained based on the acquired spatial query point and the three-dimensional skeleton splicing feature, including: Calculating the relative distance between the joint points in the three-dimensional skeleton splicing feature and the spatial query point; Detecting the correlation between the joint point and the spatial query point based on a threshold function; Obtaining a feature expression of the human skeleton based on the spatial query point based on the detection result; The priori three-dimensional feature volume is obtained based on the feature expression.

12. The method for constructing a three-dimensional human body model according to any one of claims 1 to 11, characterized in that: Constructing the three-dimensional human body model based on the normal map, the enhanced depth information and the prior three-dimensional feature volume of the human skeleton, comprising: The normal map of the monocular human body image includes a front normal map and a back normal map of the human body, wherein the front normal map and the back normal map are used to provide front and back surface information of the human body, and feature extraction is performed on the front normal map, the back normal map and the monocular human body image based on the projection points in the prior three-dimensional feature volume to obtain a two-dimensional feature; Based on the position points in the monocular human body image, sampling the two-dimensional features to obtain sampling features; The sampling feature, the enhanced depth information and the prior three-dimensional feature volume are subjected to feature fusion of the projection point to obtain a fused feature; The fusion feature is used as an additional parameter of the implicit function to solve the implicit function, so as to construct the three-dimensional human body model.

13. A three-dimensional human body model construction device, characterized in that: include: A first generating module, configured to generate a normal map of the monocular human image in response to the acquired monocular human image; A lightweight depth processing module, used for inputting the monocular human image into a lightweight depth model, so as to construct an initial cost volume of the monocular human image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human image; A second generating module is used to generate a priori three-dimensional feature volume of a human skeleton based on a parameterized model corresponding to the monocular human image; A construction module is used to construct the three-dimensional human body model based on the normal map, the enhanced depth information and the prior three-dimensional feature volume of the human skeleton.

14. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the three-dimensional human body model construction method according to any one of claims 1 to 12 by executing the executable instructions.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional human body model construction method described in any one of claims 1 to 12 is implemented.

16. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional human body model construction method described in any one of claims 1 to 12 is implemented.

Images