Model construction method, construction device, equipment, storage medium and program product

Abstract

The disclosure provides a three-dimensional human body model construction method, a construction device, equipment, a storage medium and a program product, relating to the technical field of three-dimensional modeling. Among them, the three-dimensional human body model construction method comprises: in response to the monocular human body image obtained, generating the normal map of the monocular human body image; inputting the monocular human body image into a lightweight depth model to construct an initial cost volume of the monocular human body image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human body image; generating a prior three-dimensional feature body of the human body skeleton based on the parameterized model corresponding to the monocular human body image; and constructing a three-dimensional human body model based on the normal map, the enhanced depth information and the prior three-dimensional feature body of the human body skeleton. Through the technical scheme of the disclosure, the lightweight depth model is used, and the calculation amount and the parameter quantity of the construction of the three-dimensional human body model are reduced.

Description

Technical Field

[0001] This disclosure relates to the field of 3D modeling technology, and in particular to a method for constructing a 3D human body model, a device for constructing a 3D human body model, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Single-view 3D human reconstruction refers to the reconstruction of a 3D human model using images (such as photos or video frames) from a single viewpoint. It is gradually being widely used in scenarios where service robots interact with humans and in augmented reality scenarios. The construction of a 3D human model requires the introduction of a large number of human-related features and a relatively cumbersome calculation process. While the computational complexity is high, it also causes memory consumption on computing devices. Therefore, there is an urgent need for a model construction scheme that can reduce computational complexity and memory consumption while ensuring modeling accuracy.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method for constructing a three-dimensional human body model, a device for constructing a three-dimensional human body model, an electronic device, a computer-readable storage medium, and a computer program product, which at least to some extent overcome the problems of high modeling computational complexity and memory occupation of computing devices in related technologies.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, a method for constructing a three-dimensional human body model is provided, comprising: generating a normal map of the monocular human body image in response to an acquired monocular human body image; inputting the monocular human body image into a lightweight depth model to construct an initial cost volume of the monocular human body image based on the lightweight depth model, and performing lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human body image; generating a prior three-dimensional feature volume of a human skeleton based on a parametric model corresponding to the monocular human body image; and constructing the three-dimensional human body model based on the normal map, the enhanced depth information, and the prior three-dimensional feature volume of the human skeleton.

[0007] In one embodiment of this disclosure, the lightweight depth model includes a multi-scale feature extractor, a cost body construction module, and a lightweight cost body regularization network module. The monocular human image is input into the lightweight depth model to construct an initial cost body for the monocular human image based on the lightweight depth model. The initial cost body is then subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image. This includes: inputting the monocular human image and a reference image into the multi-scale feature extractor and performing discrete wavelet transform to extract first image features at multiple depth levels of the monocular human image and second image features at multiple depth levels of the reference image; inputting the first image features and the second image features into the cost body construction module to apply affine transforms to the first image features based on the second image features at different viewpoints to construct the initial cost body; and performing lightweight regularization processing on the initial cost body based on the lightweight cost body regularization network module to obtain the enhanced depth information.

[0008] In one embodiment of this disclosure, the lightweight cost volume regularization network module is constructed based on three-dimensional depth-separable convolutional operations. The lightweight cost volume regularization network module is used to perform lightweight regularization processing on the initial cost volume to obtain the enhanced depth information, including: performing feature capture and fusion on the initial cost volume based on three-dimensional depth-separable convolutional operations to obtain a corresponding intermediate cost volume; and performing information enhancement processing on the intermediate cost volume in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the enhanced depth information.

[0009] In one embodiment of this disclosure, the three-dimensional depth-separable convolution operation includes depthwise convolution and spatial point convolution. Based on the three-dimensional depth-separable convolution operation, feature capture and fusion are performed on the initial cost volume to obtain a corresponding intermediate cost volume. This includes: independently convolving tensors of different depths in the initial cost volume into corresponding channels based on the depthwise convolution operation to obtain corresponding independent channel features, where the independent channel features include depth features corresponding to human pixels, local human structural features, and global human context features; and integrating the independent channel features in the depth direction and fusing features in local regions based on the spatial point convolution operation to obtain the intermediate cost volume of the initial cost volume at multiple depth levels.

[0010] In one embodiment of this disclosure, the intermediate cost body is subjected to information enhancement processing in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the enhanced depth information, including: configuring channel attention weights, spatial attention weights and depth attention weights for the intermediate cost body based on serial operations to obtain the enhanced depth information.

[0011] In one embodiment of this disclosure, configuring channel attention weights, spatial attention weights, and depth attention weights for the intermediate cost bodies based on serial operations to obtain the enhanced depth information includes: performing a nonlinear dimensionality reduction operation on the intermediate cost bodies at multiple depth levels based on a channel attention mechanism to obtain the channel attention weights; configuring the channel attention weights for the intermediate cost bodies to obtain a first cost body; identifying matching points in the first cost body corresponding to real human structures based on a spatial attention mechanism to obtain the spatial attention weights; configuring the spatial attention weights for the first cost body to obtain a second cost body; performing a convolution operation along the depth dimension of the second cost body based on a depth attention mechanism to obtain depth attention weights; and configuring the depth attention weights for the second cost body to obtain the enhanced depth information.

[0012] In one embodiment of this disclosure, a nonlinear dimensionality reduction operation is performed on the intermediate cost bodies of the multiple depth levels based on a channel attention mechanism to obtain channel attention weights. This includes: performing max pooling on the intermediate cost bodies to obtain first local feature information; performing average pooling on the intermediate cost bodies to obtain first global feature information; inputting the first local feature information and the first global feature information into a multilayer perceptron for aggregation to obtain aggregated features; and activating the aggregated features to obtain the channel attention weights.

[0013] In one embodiment of this disclosure, the spatial attention weights are obtained by identifying matching points in the first cost body corresponding to real human structures based on a spatial attention mechanism, including: performing max pooling on the first cost body to obtain second local feature information; performing average pooling on the first cost body to obtain second global feature information; filtering the second local feature information and the second global feature information along the spatial direction to obtain key spatial location information related to depth matching; and activating the key spatial location information to obtain the spatial attention weights.

[0014] In one embodiment of this disclosure, a convolution operation is performed along the depth dimension of the second cost body based on a deep attention mechanism to obtain deep attention weights, including: performing max pooling on the second cost body to obtain third local feature information; performing average pooling on the second cost body to obtain third global feature information; performing depth enhancement on the third local feature information and the third global feature information along the depth direction based on a specified convolution kernel, and activating the depth-enhanced information to obtain the deep attention weights.

[0015] In one embodiment of this disclosure, generating a priori three-dimensional feature volume of a human skeleton based on a parametric model corresponding to the monocular human image includes: obtaining a three-dimensional spatial layout feature of the human skeleton corresponding to the monocular human image based on the human skeleton structure in the parametric model; obtaining two-dimensional projection coordinates of the three-dimensional spatial layout feature; extracting corresponding two-dimensional skeleton features from the monocular human image based on the two-dimensional projection coordinates; stitching the two-dimensional skeleton feature and the three-dimensional spatial layout feature together to obtain a three-dimensional skeleton stitched feature; and obtaining the priori three-dimensional feature volume based on the obtained spatial query point and the three-dimensional skeleton stitched feature, wherein the spatial query point is used to query relevant nodes in the three-dimensional skeleton stitched feature.

[0016] In one embodiment of this disclosure, obtaining the prior three-dimensional feature body based on the acquired spatial query point and the three-dimensional skeleton splicing features includes: calculating the relative distance between the joints in the three-dimensional skeleton splicing features and the spatial query point; detecting the correlation between the joints and the spatial query point based on a threshold function; obtaining a feature representation of the human skeleton based on the spatial query point based on the detection result; and obtaining the prior three-dimensional feature body based on the feature representation.

[0017] In one embodiment of this disclosure, the construction of the three-dimensional human body model based on the normal map, the enhanced depth information, and the prior three-dimensional feature volume of the human skeleton includes: the normal map of the monocular human body image includes a frontal normal map and a back normal map of the human body, the frontal normal map and the back normal map are used to provide information on the front and back surfaces of the human body; feature extraction is performed on the frontal normal map, the back normal map, and the monocular human body image based on the projection points in the prior three-dimensional feature volume to obtain two-dimensional features; the two-dimensional features are sampled based on the position points in the monocular human body image to obtain sampled features; the sampled features, the enhanced depth information, and the prior three-dimensional feature volume are fused using the features of the projection points to obtain fused features; and the fused features are used as additional parameters of an implicit function to solve the implicit function to construct the three-dimensional human body model.

[0018] According to another aspect of this disclosure, a three-dimensional human body model construction apparatus is provided, comprising: a first generation module, configured to generate a normal map of the monocular human body image in response to an acquired monocular human body image; a lightweight depth processing module, configured to input the monocular human body image into a lightweight depth model to construct an initial cost volume of the monocular human body image based on the lightweight depth model, and to perform lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human body image; a second generation module, configured to generate a prior three-dimensional feature volume of a human skeleton based on a parameterized model corresponding to the monocular human body image; and a construction module, configured to construct the three-dimensional human body model based on the normal map, the enhanced depth information, and the prior three-dimensional feature volume of the human skeleton.

[0019] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor being configured to perform the above-described three-dimensional human body model construction method by executing the executable instructions.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for constructing a three-dimensional human body model.

[0021] According to another aspect of this disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for constructing a three-dimensional human body model.

[0022] The three-dimensional human body model construction scheme provided by the embodiments of this disclosure reduces the amount of computation and the number of parameters by using a lightweight depth model. This ensures the normal execution of the three-dimensional human body model construction without occupying a large amount of device memory. The regularization processing of the cost volume can reduce the impact of noise and outliers, improve the quality of the cost volume, and enable the obtained enhanced depth information to improve the accuracy of depth estimation. The combination of the normal map provides geometric information of the image surface and prior knowledge of human body structure provided by the prior three-dimensional feature volume of the human skeleton. It does not require storing a large amount of intermediate data. While reducing the high computational complexity and memory occupation in the construction of the three-dimensional human body model, it improves the accuracy, robustness and computational efficiency of the constructed three-dimensional human body model.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 This diagram illustrates a three-dimensional human body model construction system according to an embodiment of the present disclosure;

[0026] Figure 2 A schematic flowchart of a three-dimensional human body model construction method according to an embodiment of the present disclosure is shown;

[0027] Figure 3 A schematic flowchart of another three-dimensional human body model construction method in an embodiment of this disclosure is shown;

[0028] Figure 4 This diagram illustrates an embodiment of encoding based on three-dimensional depthwise separable convolution.

[0029] Figure 5 A schematic flowchart of another three-dimensional human body model construction method in an embodiment of this disclosure is shown;

[0030] Figure 6 This diagram illustrates a decoding method based on a three-dimensional hybrid attention mechanism according to an embodiment of the present disclosure.

[0031] Figure 7 A schematic flowchart of another three-dimensional human body model construction method in this disclosure is shown;

[0032] Figure 8 This diagram illustrates another three-dimensional human body model construction device according to an embodiment of the present disclosure;

[0033] Figure 9 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] Single-view 3D human reconstruction technology has been applied in some cases. For example, when service robots interact with people, they need to estimate the 3D human model of the interacting object from their own perspective; augmented reality applications also need to estimate the 3D model of the target human body from their own perspective.

[0037] From a visual perspective, since single-view images can only provide visual information about the front of the human body, the 3D reconstruction process relies heavily on prior knowledge of human geometry. In some embodiments, an RGB-D camera is a camera that can simultaneously acquire color (RGB) image information and depth information. It can match the statistically parameterized human body model with the RGB-D image to reconstruct the 3D model of the human body. However, if the human body is partially occluded by an object in the image acquired by the RGB-D camera, the visible surface area of ​​the human body is greatly reduced, which will lead to a decrease in the fitting accuracy of the model. Furthermore, the image information of the occluded object may also introduce interference, resulting in a decrease in the 3D reconstruction effect.

[0038] To address the aforementioned shortcomings, this disclosure introduces a lightweight deep model and incorporates cost volume regularization to reduce the model's computational complexity, effectively decreasing the number of parameters and computational resource consumption.

[0039] To facilitate understanding, the following is an explanation of several terms used in this application.

[0040] Normal map: It is an image that represents the orientation information of the surface of an object. In three-dimensional space, each point on the surface of the object has a normal vector, which is perpendicular to the surface where the point is located. The normal map usually uses color coding to represent the direction of the normal vector. For example, the three channels in the RGB color space can be used to represent the components of the normal vector on the three coordinate axes respectively.

[0041] Cost volume: A data structure used to represent the matching cost under different assumptions. In computer vision, it is often used to solve problems such as stereo matching and depth estimation. The cost volume is usually a three-dimensional or higher-dimensional array, where each element represents the matching cost between two images or features under a specific assumption (such as different depth values, disparity, etc.).

[0042] Regularization is a technique used during model training and optimization to prevent overfitting and improve the model's generalization ability. Regularization is usually achieved by adding a penalty term to the objective function, which is related to the complexity of the model.

[0043] Affine transformation is a linear transformation that can perform operations such as translation, rotation, scaling, and shearing on points, vectors, and graphics in two-dimensional or three-dimensional space.

[0044] Figure 1 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application. The system includes: several tenant terminals 120 and a server terminal 140.

[0045] Tenant terminal 120 can be a mobile terminal such as a mobile phone, game console, tablet computer, e-book reader, smart glasses, MP4 (Moving Picture Experts Group Audio Layer IV) player, smart home device, AR (Augmented Reality) device, VR (Virtual Reality) device, etc. Alternatively, tenant terminal 120 can also be a personal computer (PC), such as a laptop computer and a desktop computer, etc.

[0046] Among them, the tenant terminal 120 may have an application installed for providing 3D human body model construction.

[0047] The tenant 120 and the server 140 are connected via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0048] Server 140 is a single server, or a combination of several servers, or a virtualization platform, or a cloud computing service center. Server 140 provides backend services for applications that construct 3D human models. Optionally, server 140 undertakes the primary computational work, and tenant 120 undertakes secondary computational work; or, server 140 undertakes secondary computational work, and tenant 120 undertakes primary computational work; or, tenant 120 and server 140 collaborate on computation using a distributed computing architecture.

[0049] In some alternative embodiments, server 140 is used to store information about the three-dimensional human body model construction program.

[0050] Optionally, the logistics user terminals of the applications installed on different tenant terminals 120 are the same, or the logistics user terminals of the applications installed on two tenant terminals 120 are the same type of application logistics user terminals on different control system platforms. Depending on the terminal platform, the specific form of the application's logistics user terminal can also differ; for example, the application's logistics user terminal can be a mobile logistics user terminal, a PC logistics user terminal, or a World Wide Web (Web) logistics user terminal, etc.

[0051] Those skilled in the art will understand that the number of the aforementioned tenant terminals 120 can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0052] Optionally, the system may also include a management device ( Figure 1 (Not shown), the management device is connected to the server 140 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0053] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0054] like Figure 2 As shown, a three-dimensional human body model construction method according to an embodiment of the present disclosure includes:

[0055] Step S202: In response to the acquired monocular human image, generate the normal map of the monocular human image.

[0056] In some embodiments, the normal map of a monocular human body image includes a frontal normal map and a back normal map. The frontal and back normal maps can describe the geometry of the human body. The normal map provides information on the geometry of the human body surface from different perspectives. Each pixel in the normal map represents the normal direction of the corresponding surface point, i.e., a vector perpendicular to the surface. By analyzing the normal map, the curvature, concavity, convexity, and other geometric features of the human body surface can be inferred.

[0057] In some embodiments, the normal map can also be used to determine details of the human body surface, such as skin texture and clothing wrinkles.

[0058] In some embodiments, the frontal and back normal maps can provide clues for depth estimation of monocular human images, and serve as constraints to constrain the depth estimation process of monocular human images.

[0059] In some embodiments, the frontal and back normal maps can be combined with other features (such as two-dimensional image features, skeletal structure prior information, etc.) to jointly construct a three-dimensional human body model. By fusing multiple features, the performance of model construction can be improved.

[0060] Step S204: Input the monocular human image into the lightweight depth model to construct the initial cost volume of the monocular human image based on the lightweight depth model, and perform lightweight regularization processing on the initial cost volume to obtain the enhanced depth information of the monocular human image.

[0061] In some embodiments, a lightweight neural network architecture is used to build a lightweight deep model, such as a lightweight deep model based on MobileNet or ShuffleNet.

[0062] In some embodiments, the cost body is a three-dimensional data structure, which can be represented as D×H×W, where D represents the depth dimension, and H and W represent the height and width of the image, respectively.

[0063] In some embodiments, optimizing the initial cost volume through lightweight regularization can reduce the impact of noise and outliers, thereby obtaining more accurate depth information in human images.

[0064] Step S206: Generate a priori 3D feature volume of the human skeleton based on the parametric model corresponding to the monocular human image.

[0065] In some embodiments, the parametric model is used to provide shape and posture parameters of the human body, as well as corresponding body feature representations. The parametric model may be SMPL (Skinned Multi-Person Linear Model), etc.

[0066] In some embodiments, the parameterized model can also be used to determine additional normal prior information.

[0067] In some embodiments, the joint positions and connections of the human body can be determined based on the human skeleton information in the parametric model, and a three-dimensional feature body can be constructed based on this information, wherein each voxel can include feature information related to the human skeleton.

[0068] Step S208: Construct a three-dimensional human body model based on the normal map, enhanced depth information, and prior three-dimensional feature volume of the human skeleton.

[0069] In some embodiments, implicit functions, mesh reconstruction, and other methods can be used to construct a three-dimensional human body model by integrating information from the normal map, enhanced depth information, and prior three-dimensional feature bodies of the human skeleton.

[0070] In this embodiment, by using a lightweight depth model, the computational load and number of parameters are reduced, ensuring the normal execution of the 3D human body model construction without requiring a large amount of device memory. Regularization of the cost volume can reduce the impact of noise and outliers, improve the quality of the cost volume, and enable the obtained enhanced depth information to improve the accuracy of depth estimation. The combination of the normal graph provides geometric information of the image surface and the prior knowledge of human structure provided by the prior 3D feature volume of the human skeleton. It does not require storing a large amount of intermediate data. While reducing the high computational complexity and memory consumption in the construction of the 3D human body model, it improves the accuracy, robustness and computational efficiency of the constructed 3D human body model.

[0071] like Figure 3 As shown, in one embodiment of this disclosure, the lightweight depth model includes a multi-scale feature extractor, a cost volume construction module, and a lightweight cost volume regularization network module. A monocular human image is input into the lightweight depth model to construct an initial cost volume of the monocular human image based on the lightweight depth model. The initial cost volume is then subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image, including:

[0072] Step S302: Input the monocular human image and the reference image into the multi-scale feature extractor and perform discrete wavelet transform to extract the first image features of multiple depth levels of the monocular human image and the second image features of multiple depth levels of the reference image.

[0073] Among them, the multi-scale feature extractor has an embedded discrete wavelet transform function, which feeds the source image and the reference image into the multi-scale feature extractor to extract multi-level image features.

[0074] In some embodiments, a discrete wavelet transform is performed using a reference image as a comparison benchmark to decompose the image into sub-bands of different frequencies, thereby extracting features at different scales. By using the reference image as a benchmark, the differences and similarities between the monocular human image and the reference image can be captured better.

[0075] Step S304: Input the first image features and the second image features into the cost body construction module, and apply affine transformation to the first image features based on the second image features under different viewpoints to construct the initial cost body.

[0076] In some embodiments, the cost body construction module is used to apply affine transformations to the first image features based on the second image features from different perspectives to construct an initial cost body. By applying affine transformations to the first image features, image changes from different perspectives can be simulated to construct the initial cost body.

[0077] Step S306: The initial cost volume is subjected to lightweight regularization processing based on the lightweight cost volume regularization network module to obtain enhanced depth information.

[0078] In some embodiments, the lightweight cost volume regularization network module can employ lightweight neural network structures, such as depthwise separable convolutions or bottleneck structures, to optimize the initial cost volume and obtain enhanced depth information, thereby reducing computational load and the number of parameters.

[0079] In this embodiment, the multi-scale feature extractor can extract image features from different depth levels, thereby better capturing the details and structure of the image. In the cost body construction module, by applying affine transformation to the first image features, the image changes under different perspectives can be simulated, thereby constructing a more accurate cost body. Furthermore, the lightweight cost body regularization network module adopts a lightweight neural network structure, which can reduce the amount of computation and the number of parameters, and improve computational efficiency.

[0080] In one embodiment of this disclosure, a lightweight cost volume regularization network module is constructed based on three-dimensional depth-separable convolutional operations. The initial cost volume is subjected to lightweight regularization processing based on the lightweight cost volume regularization network module to obtain enhanced depth information, including:

[0081] Based on three-dimensional depth-separable convolution operations, features are captured and fused from the initial cost volume to obtain the corresponding intermediate cost volume.

[0082] Among them, 3D depth separable convolution refers to decomposing 3D convolution into convolution in the depth direction and convolution in the spatial direction, which helps to reduce the amount of computation and the number of parameters.

[0083] In some embodiments, the process of performing lightweight regularization processing on the initial cost volume based on the lightweight cost volume regularization network module to obtain enhanced depth information may include an encoding stage and a decoding stage. The encoding stage is characterized by performing feature capture and fusion on the initial cost volume based on three-dimensional depth separable convolution operations to obtain the corresponding intermediate cost volume.

[0084] In some embodiments, during the encoding process, when capturing features in the initial cost volume, convolution in the depth direction focuses on extracting features at different depth levels, while convolution in the spatial direction captures the spatial structure information of the image. This effectively extracts key features from the initial cost volume and fuses them. After the three-dimensional depth-separable convolution operation, the corresponding intermediate cost volume is obtained. The intermediate cost volume includes information after preliminary feature extraction and fusion, providing a foundation for the subsequent decoding process.

[0085] Based on a three-dimensional hybrid attention mechanism, information enhancement processing is performed on the intermediate cost volume in different dimensions to obtain enhanced depth information.

[0086] In some embodiments, during the decoding phase of the regularization network, in order to reduce the impact of feature matching on cost volume regularization and ensure its accuracy, a 3D hybrid attention module can enhance important feature information and suppress the influence of anomalous data, remove noise and unnecessary data, thereby generating enhanced depth information.

[0087] In this embodiment, a lightweight cost body regularization network module is constructed using three-dimensional depthwise separable convolution operations. Depthwise separable convolution greatly reduces computational complexity, enabling efficient depth information enhancement processing to obtain intermediate cost bodies even on resource-limited devices. While reducing computational complexity, the accuracy of regularization is ensured. Furthermore, by using a three-dimensional hybrid attention mechanism to enhance information processing on the intermediate cost body in different dimensions, key features of depth information can be captured more effectively, enhancing the robustness of the module.

[0088] In one embodiment of this disclosure, the three-dimensional depth-separable convolution operation includes depthwise convolution and spatial point convolution. Based on the three-dimensional depth-separable convolution operation, feature capture and fusion are performed on the initial cost volume to obtain a corresponding intermediate cost volume, including:

[0089] Based on depthwise convolution, tensors of different depths in the initial cost volume are input into the corresponding channels for independent convolution to obtain the corresponding independent channel features. The independent channel features include depth features corresponding to human pixels, local structural features of the human body, and global context features of the human body.

[0090] In some embodiments, depthwise convolution is performed on each channel of the input tensor, with each channel having an independent convolution kernel, without involving cross-channel convolution. That is, the convolution kernel is applied independently to each channel of the input tensor, which can not only significantly reduce the number of parameters and computational complexity, but also allow the network to learn richer feature representations without increasing the computational burden too much.

[0091] In some embodiments, for an input tensor of size C in ×H×W(C in (where H and W are the height and width, respectively). When operating with a depthwise convolution kernel of size K×K×K (K represents the kernel size), the number of output channels is C. out The number of parameters is C in ×K×K×K, compared to the number of parameters C in a non-separable 3D convolution. in ×K×K×K×C out This greatly reduces [the number of cases].

[0092] Based on spatial point convolution operations, the independent channel features are integrated in the depth direction and the features of local regions are fused to obtain the intermediate cost volume of the initial cost volume at multiple depth levels.

[0093] In some embodiments, after depthwise convolution, pointwise convolution combines the output of each channel. By applying a convolution kernel to each local region, pointwise convolution effectively integrates feature information in the depth direction. It can not only capture local features in the spatial dimension, but also understand the distribution and changes of these features in the depth direction. It applies pointwise convolution of 3D depth to feature maps that are independent of channels, thereby achieving the aggregation of channel information.

[0094] like Figure 4 As shown, the cost volume includes the dimension D in the depth direction, the dimension H in the height direction, and the dimension W in the width direction. It is processed by a 3×3×1 convolution kernel. That is, when performing depth dimension convolution, the convolution operation mainly extracts features in the height and width directions using a 3×3 window, and processes each slice in the depth direction.

[0095] The convolution kernel of spatial point convolution is 1×1×3. It mainly extracts features around a spatial point in the depth direction with a 3-dimensional window, while operating on a single point in the height and width directions. This allows for the extraction and integration of feature information from different angles, providing more effective feature representations for subsequent tasks such as 3D human body model construction.

[0096] In this embodiment, a complete convolution operation is formed by sequentially executing depthwise convolution and spatial point convolution. Depthwise convolution is used for feature decomposition, and spatial point convolution is used for feature recombination. 3D depthwise separable convolution not only reduces the number of model parameters and computational complexity, but also maintains or even improves the feature extraction capability. It can help lightweight deep models learn the nonlinear relationships between different channels. In the lightweight cost volume regularization network module, 3D depthwise separable convolution is used to extract and fuse cost volume features from different perspectives, which is beneficial to significantly reduce the number of parameters and computational complexity, enabling the network to process large-scale 3D data more efficiently. At the same time, due to its deep separation processing of features, the regularization network can better capture local details and global context information.

[0097] In one embodiment of this disclosure, information augmentation processing is performed on the intermediate cost volume in different dimensions based on a three-dimensional hybrid attention mechanism to obtain enhanced depth information, including:

[0098] Based on serial operations, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body to obtain enhanced depth information.

[0099] In this context, serial operation refers to applying the weights generated in the previous step to the next step of processing.

[0100] In some embodiments, the channel attention mechanism focuses on the importance of different channels in the intermediate cost volume. Different channels may represent different feature information, such as human body appearance color, texture and edge shape. By calculating the channel attention weights, it can automatically learn which channels are more critical for depth estimation, thereby enabling targeted enhancement or suppression of different channels.

[0101] Spatial attention mechanisms emphasize the importance of different spatial locations in the intermediate cost volume. Different spatial locations may have different depth information values. Spatial attention weights can highlight key spatial locations that may correspond to the real human body geometry, while suppressing less important spatial locations or those that may be affected by noise.

[0102] The deep attention mechanism focuses on the similarity confidence at different depth levels of the intermediate cost body. Different depth hypotheses correspond to different depth levels. The deep attention weights can adjust the confidence of different depth levels, enhance the depth levels that are more likely to occur, and suppress the depth levels that are less likely to occur.

[0103] In this embodiment, the intermediate cost volume is weighted sequentially by attention mechanisms in three dimensions: channel, space, and depth. This can more accurately highlight features that make important contributions to depth estimation. Different attention mechanisms focus on key information from different angles, thereby reducing interference from irrelevant or noisy information and improving the accuracy of depth information.

[0104] like Figure 5 As shown, in one embodiment of this disclosure, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body based on serial operations to decode enhanced depth information, including:

[0105] Step S502: Perform nonlinear dimensionality reduction on intermediate cost volumes at multiple depth levels based on the channel attention mechanism to obtain channel attention weights.

[0106] In one embodiment of this disclosure, a nonlinear dimensionality reduction operation is performed on intermediate cost bodies at multiple depth levels based on a channel attention mechanism to obtain channel attention weights. This includes: performing max pooling on the intermediate cost bodies to obtain first local feature information; performing average pooling on the intermediate cost bodies to obtain first global feature information; inputting the first local feature information and the first global feature information into a multilayer perceptron for aggregation to obtain aggregated features; and activating the aggregated features to obtain channel attention weights.

[0107] In some embodiments, the channel attention submodule is responsible for performing attention operations on the channel information in the cost body. First, it performs a nonlinear dimensionality reduction operation on the intermediate cost body, with the dimensionality reduction coefficient set to 16, to obtain the channel attention weights W. 1 The channel attention weights are obtained by performing max pooling and average pooling operations on the cost volume, then aggregating it through a convolutional layer, and finally applying an activation function, as shown in Equation (1).

[0108] W 1 =σ(M(MP(V))+M(AP(V))) (1)

[0109] Among them, W 1 σ represents the channel attention weight, M represents the multilayer perceptron, MP represents max pooling, and AP represents average pooling.

[0110] Step S504: Configure channel attention weights for the intermediate cost body to obtain the first cost body.

[0111] In some embodiments, the channel attention weight W 1 Multiply by the intermediate cost volume V to obtain the first cost volume V after channel refinement. 1 As shown in equation (2).

[0112] V 1 =V×W 1 (2)

[0113] Step S506: Based on the spatial attention mechanism, identify the matching points in the first cost body that correspond to the real human body structure, and obtain the spatial attention weights.

[0114] In one embodiment of this disclosure, the spatial attention mechanism is used to identify matching points in the first cost volume corresponding to the real human body structure, and to obtain spatial attention weights for enhancing the occluded region. This includes: performing max pooling on the first cost volume to obtain second local feature information; performing average pooling on the first cost volume to obtain second global feature information; filtering the second local feature information and the second global feature information along the spatial direction to enhance the depth representation of the occluded region in the monocular human image to obtain key spatial location information related to depth matching; and activating the key spatial location information to obtain spatial attention weights.

[0115] In some embodiments, the spatial attention submodule can effectively perceive cost volume information at different spatial locations. By using the spatial attention mechanism, it can identify the matching point most likely to correspond to the real geometric structure, thereby improving the accuracy of matching. Here, a convolutional kernel of size 1×7×7 is used to filter the first cost volume information along the spatial direction, which enhances the feature representation while reducing noise interference, so that the convolutional kernel can cover a larger spatial range, thereby capturing richer spatial context information. This enables more accurate identification and emphasis of key spatial locations related to depth matching, while suppressing less important information or information that may be caused by noise, providing accurate spatial information for subsequent processing stages. The corresponding spatial attention weights are shown in Equation (3).

[0116] W 2 =σ(conv) 1×7×7 (MP(V 1 )+AP(V 1 (3)

[0117] Among them, W 2 Let σ represent the spatial attention weights, and conv be the activation function. 1×7×7 represents spatial convolution, MP represents max pooling, and AP represents average pooling.

[0118] Step S508: Configure spatial attention weights for the first cost body to obtain the second cost body.

[0119] Among them, the second cost body V is calculated. 2 As shown in equation (4).

[0120] V 2 =V 1×W 2 (4)

[0121] Step S510: Perform convolution operation along the depth dimension of the second cost body based on the deep attention mechanism to obtain the deep attention weights.

[0122] In one embodiment of this disclosure, a convolution operation is performed along the depth dimension of the second cost body based on a deep attention mechanism to obtain deep attention weights, including: performing max pooling on the second cost body to obtain third local feature information; performing average pooling on the second cost body to obtain third global feature information; performing depth enhancement on the third local feature information and the third global feature information along the depth direction based on a specified convolution kernel, and activating the depth-enhanced information to obtain deep attention weights.

[0123] In some embodiments, to improve the accuracy of matching information for different depth hypotheses at the same spatial location in 3D reconstruction, the depth domain attention module operates along the depth dimension of the cost volume using a 7×1×1 convolutional kernel, thereby enhancing the depth information. This enables the model to capture and distinguish features at different depths more precisely, effectively enhancing the network's ability to recognize and process depth information, and thus improving the accuracy of matching and the overall quality of reconstruction. The corresponding depth attention weights are shown in Equation (5). Finally, these three modules are cascaded and combined to form a 3D hybrid attention module.

[0124] W 3 =σ(conv) 7×1×1 (MP(V 2 )+AP(V 2 ))) (5)

[0125] Step S512: Configure depth attention weights for the second cost body to obtain enhanced depth information.

[0126] Among them, the enhanced depth information V is calculated. 3 As shown in equation (6).

[0127] V 3 =V 2 ×W 3 (6)

[0128] like Figure 6 As shown, the channel attention submodule is responsible for performing attention operations on the channel information in the cost volume, the spatial attention submodule can effectively perceive the cost volume information at different spatial locations, and the depth domain attention module operates along the depth dimension of the cost volume by using convolution kernels, thereby enhancing the depth information.

[0129] In this embodiment, by combining the effects of channel attention, spatial attention, and depth attention, the model can comprehensively enhance its attention to information in different dimensions of the cost volume. By combining the attention weights of these three dimensions, the model can generate a final cost volume tensor that integrates multiple aspects of information, that is, enhances depth information and suppresses the influence of abnormal data on depth information. This is conducive to optimizing the regularization process of the cost volume and improving the accuracy of 3D human feature matching.

[0130] In one embodiment of this disclosure, generating a priori 3D feature volume of the human skeleton based on a parametric model corresponding to a monocular human image includes:

[0131] Based on the human skeleton structure in the parametric model, the three-dimensional spatial layout features of the human skeleton corresponding to the monocular human image are obtained; the two-dimensional projection coordinates of the three-dimensional spatial layout features are obtained; the corresponding two-dimensional skeleton features are extracted from the monocular human image based on the two-dimensional projection coordinates; the two-dimensional skeleton features and the three-dimensional spatial layout features are stitched together to obtain the three-dimensional skeleton stitched features; the prior three-dimensional feature volume is obtained based on the obtained spatial query points and the three-dimensional skeleton stitched features, and the spatial query points are used to query relevant nodes in the three-dimensional skeleton stitched features.

[0132] In some embodiments, human-related information is input into a parametric model. This monocular human image includes the human's appearance features, posture information, etc. The parametric model generates three-dimensional spatial layout features of the human skeleton based on the input data, including the position of each joint, the length and direction of the bones, etc. Based on the three-dimensional spatial layout features, the shape and structure of the human skeleton in three-dimensional space can be determined.

[0133] Furthermore, based on the generated 3D spatial layout information, the projected coordinates of each skeleton point on the 2D plane are calculated. Using the calculated 2D projected coordinates, features corresponding to the skeleton points are extracted from the corresponding monocular human body image. The extracted 2D features are then stitched together with the original 3D features. Stitching can be achieved through a simple concatenation operation, combining the 2D and 3D features in a certain order to form a richer feature vector. The stitched feature vector is then fed into a multilayer perceptron, which determines whether a specific sampling point is located inside or outside the human body based on the input feature vector.

[0134] In this embodiment, a three-dimensional feature map is generated based on a parametric model to obtain the three-dimensional spatial layout information of the skeleton. Next, using the two-dimensional projection coordinates of this three-dimensional information, corresponding features are extracted from the corresponding two-dimensional feature map. To enhance the richness and accuracy of the features, the extracted two-dimensional features are concatenated with the original three-dimensional features. These concatenated features are then fed into a multilayer perceptron to determine whether a specific sampling point is located inside or outside the human body, thereby effectively improving the ability to recognize human skeleton features.

[0135] In one embodiment of this disclosure, a priori 3D feature volume is obtained based on the acquired spatial query points and 3D skeleton splicing features, including:

[0136] Calculate the relative distance between joints and spatial query points in the 3D skeleton splicing features; detect the correlation between joints and spatial query points based on a threshold function; obtain the feature representation of the human skeleton based on spatial query points based on the detection results; and obtain the prior 3D feature volume based on the feature representation.

[0137] In some embodiments, the spatial query point can be any point selected in three-dimensional space, with the purpose of calculating the relative position with the key points to extract features about the human skeleton. For each key point and spatial query point, the relative distance between them is calculated, which can be done using the Euclidean distance formula.

[0138] In some embodiments, the feature representation of the human skeleton based on spatial query points is shown in equations (7) to (9).

[0139] r k =[r k,1 ,……,r k,m (7)

[0140] r k，j =G(‖p k -J j ||2∈R), j∈(1,m) (8)

[0141] Where, r k p represents the k-th spatial query point k Relative to the feature representation of each key point, m refers to the number of key points, k refers to the number of spatial query points, and G(‖p k -J j ||2∈R) represents the threshold function. By calculating the relative position of the control query point with each joint, the feature expression of the spatial query point relative to each joint can be obtained. This is achieved by introducing the threshold function G(||p) k -J j ||2∈R) can reduce the influence of irrelevant nodes in feature representation.

[0142] The prior information of the skeleton-based structure is shown in Equation (9):

[0143] H J (p k ) = r k ,k=1,...,N+K (9)

[0144] Wherein, N+K represents the spatial query points obtained by the skeleton-based sampling strategy.

[0145] In one embodiment of this disclosure, a three-dimensional human body model is constructed based on prior three-dimensional features of the normal map, enhanced depth information, and human skeleton, including:

[0146] The normal map of a monocular human image includes the frontal normal map and the back normal map of the human body. The frontal and back normal maps are used to provide information about the front and back surfaces of the human body. Based on the projection points in the prior 3D feature volume, features are extracted from the frontal normal map, the back normal map, and the monocular human image to obtain 2D features. Based on the position points in the monocular human image, the 2D features are sampled to obtain sampled features. The sampled features, enhanced depth information, and the features of the projection points of the prior 3D feature volume are fused to obtain fused features. The fused features are used as additional parameters of the implicit function to solve the implicit function to construct a 3D human model.

[0147] In some embodiments, the parameters of a parametric model can be used to generate a surface mesh of the human body. The surface mesh can be viewed as a three-dimensional object composed of many small triangles, which can accurately represent the surface shape of the human body. For the generated surface mesh, the normal vector of each triangle facet on the front and back sides is calculated separately. The normal vector is a vector perpendicular to the triangle facet and represents the direction of the surface at that point. In the case of occlusion, a monocular human body image may not be able to completely display all parts of the human body. The front normal map and the back normal map can provide geometric information of the human body surface in different directions, so as to better understand the shape and position of the occluded parts when building a three-dimensional human body model.

[0148] In some embodiments, the acquired front and back normal images of the human body and the prior structural information of the skeleton are input into the model along with two-dimensional image features. The model extracts key features that accurately reflect the human body's morphology through effective information fusion. Finally, implicit functions are used to decode these features, thereby generating a detailed three-dimensional human body model.

[0149] The pixel alignment implicit function of the human body 3D reconstruction method based on geometric prior information is shown in Equation (10).

[0150]

[0151] Here, X can be understood as a point or positional information in a 3D human body model, x = π(X) is the projection of X onto a 2D image plane, and F L B L These are the front and back normal maps, respectively. I is the input 2D monocular human image, Φ is the sampling function, which obtains the feature corresponding to projection point x on the acquired feature map based on the projection point x, and H... j (p k ) represents the structural prior information based on the skeleton, Z represents the depth value in the coordinate space, and g is the feature extraction method used by the convolutional neural network to obtain two-dimensional features.

[0152] Furthermore, the sampling features, enhanced depth information, and prior 3D feature volumes are fused by projecting points to obtain fused features. Combined with the implicit function reconstruction module, the complete reconstruction of the 3D human body is completed. While making full use of the prior knowledge of the front and back normal maps and skeleton, the accurate conversion from 2D image to 3D model is achieved through feature aggregation and implicit function decoding. By utilizing the constraints provided by multi-attribute priors, the reconstruction process of the 3D human body can be effectively standardized, thereby significantly improving the detail expression ability of the 3D human body model.

[0153] like Figure 7 As shown, according to another embodiment of the present disclosure, a three-dimensional human body model construction scheme is implemented by inputting a monocular human body image into a normal map generation module 702, a lightweight depth model 704, and a parametric model estimation module 706. The normal map generation module 702 outputs a normal map, and the parametric model estimation module 706 outputs a priori three-dimensional feature volume of the human skeleton. The lightweight depth model 704 includes a multi-scale wavelet convolution feature extraction module 7042, a cost volume construction module 7044, and a lightweight cost volume regularization module 7046, and outputs enhanced depth information. The normal map, enhanced depth information, and priori three-dimensional feature volume of the human skeleton are input into a feature aggregation module 708 to obtain aggregated features, which are used as additional parameters of the implicit function 710, thereby realizing the accurate conversion from a two-dimensional image to a three-dimensional model.

[0154] In this embodiment, the input 2D image is encoded with enhanced depth information through a lightweight depth model. The predicted parameterized model is then converted into a prior 3D feature volume of the skeleton structure through a 3D convolutional neural network. In addition, the front and back normal maps obtained by the normal map generation module are used together with the prior 3D feature volume of the skeleton structure and the 2D image features including enhanced depth information as input, and further integrated through feature aggregation technology. Finally, the implicit function reconstruction module completes the full reconstruction of the 3D human body. By utilizing the constraints provided by the multi-attribute prior, the reconstruction process of the 3D human body is effectively standardized, and the detail expression capability of the 3D human body model is significantly improved.

[0155] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0156] The following reference Figure 8 To describe a three-dimensional human body model construction apparatus 800 according to an embodiment of the present disclosure. Figure 8 The three-dimensional human body model construction device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments disclosed herein.

[0157] The 3D human body model construction device 800 is manifested in the form of hardware modules or software modules. The components of the 3D human body model construction device 800 may include, but are not limited to: a first generation module 802, used to generate a normal map of a monocular human body image in response to an acquired monocular human body image; a lightweight depth processing module 804, used to input the monocular human body image into a lightweight depth model to construct an initial cost volume of the monocular human body image based on the lightweight depth model, and to perform lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human body image; a second generation module 806, used to generate a priori 3D feature volume of the human skeleton based on a parametric model corresponding to the monocular human body image; and a construction module 808, used to construct a 3D human body model based on the normal map, enhanced depth information, and the priori 3D feature volume of the human skeleton.

[0158] like Figure 9 As shown, the electronic device 900 is manifested in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0159] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform actions such as... Figure 2 The scheme described in [the document].

[0160] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0161] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0162] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0163] Electronic device 900 can also communicate with one or more external devices 790 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or electronic device, etc.) to execute the methods according to the embodiments of this disclosure.

[0165] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon a program product capable of implementing the methods described above. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0166] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0167] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0169] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0170] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on an IoT terminal, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0171] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0172] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0173] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or electronic device, etc.) to execute the method according to the embodiments of this disclosure.

[0174] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for constructing a three-dimensional human body model, characterized in that, include: In response to the acquired monocular human image, a normal map of the monocular human image is generated; The monocular human image is input into a lightweight depth model comprising a multi-scale feature extractor, a cost volume construction module, and a lightweight cost volume regularization network module. An initial cost volume of the monocular human image is constructed based on the lightweight depth model, and the initial cost volume is then subjected to lightweight regularization processing to obtain enhanced depth information of the monocular human image. This process includes: inputting the monocular human image and a reference image into the multi-scale feature extractor and performing discrete wavelet transform to extract first image features at multiple depth levels of the monocular human image and second image features at multiple depth levels of the reference image; inputting the first image features and the second image features into the cost volume construction module to apply affine transforms to the first image features based on the second image features at different viewpoints to construct the initial cost volume; and performing lightweight regularization processing on the initial cost volume based on the lightweight cost volume regularization network module to obtain the enhanced depth information. A priori three-dimensional feature body of the human skeleton is generated based on the parameterized model corresponding to the monocular human image. The three-dimensional human model is constructed based on the normal map, the enhanced depth information, and the prior three-dimensional feature volume of the human skeleton.

2. The method for constructing a three-dimensional human body model according to claim 1, characterized in that, The lightweight cost volume regularization network module is constructed based on 3D depth-separable convolution operations. The initial cost volume is subjected to lightweight regularization processing based on the lightweight cost volume regularization network module to obtain the enhanced depth information, including: The initial cost volume is subjected to feature capture and fusion based on three-dimensional depth separable convolution operations to obtain the corresponding intermediate cost volume; The intermediate cost volume is augmented in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the augmented depth information.

3. The method for constructing a three-dimensional human body model according to claim 2, characterized in that, The 3D depth-separable convolutional operation includes depthwise convolution and spatial point convolution. Based on the 3D depth-separable convolutional operation, feature capture and fusion are performed on the initial cost volume to obtain the corresponding intermediate cost volume, including: Based on the depthwise convolution operation, tensors of different depths in the initial cost volume are input into the corresponding channels for independent convolution to obtain the corresponding independent channel features. The independent channel features include depth features corresponding to human pixels, local human structural features, and global human context features. Based on spatial point convolution operations, the independent channel features are integrated in the depth direction and the features of local regions are fused to obtain the intermediate cost body of the initial cost body at multiple depth levels.

4. The method for constructing a three-dimensional human body model according to claim 2, characterized in that, The intermediate cost volume is augmented in different dimensions based on a three-dimensional hybrid attention mechanism to obtain the augmented depth information, including: Based on serial operations, channel attention weights, spatial attention weights, and depth attention weights are configured for the intermediate cost body to obtain the enhanced depth information.

5. The method for constructing a three-dimensional human body model according to claim 4, characterized in that, Configure channel attention weights, spatial attention weights, and depth attention weights for the intermediate cost body based on serial operations to obtain the enhanced depth information, including: Based on the channel attention mechanism, a nonlinear dimensionality reduction operation is performed on the intermediate cost volumes of the multiple depth levels to obtain the channel attention weights; Configure the channel attention weights on the intermediate cost body to obtain the first cost body; Based on the spatial attention mechanism, the matching points in the first cost body corresponding to the real human body structure are identified to obtain the spatial attention weights; Configure the spatial attention weights on the first cost body to obtain the second cost body; Based on the deep attention mechanism, a convolution operation is performed along the depth dimension of the second cost body to obtain the deep attention weights; The depth attention weights are configured for the second cost body to obtain the enhanced depth information.

6. The method for constructing a three-dimensional human body model according to claim 5, characterized in that, Based on the channel attention mechanism, a nonlinear dimensionality reduction operation is performed on the intermediate cost volumes of the multiple depth levels to obtain the channel attention weights, including: Max pooling is performed on the intermediate cost body to obtain the first local feature information; The intermediate cost body is subjected to average pooling to obtain the first overall feature information; The first local feature information and the first global feature information are respectively input into a multilayer perceptron for aggregation to obtain aggregated features; The aggregated features are activated to obtain the channel attention weights.

7. The method for constructing a three-dimensional human body model according to claim 5, characterized in that, Based on the spatial attention mechanism, matching points corresponding to real human structures in the first cost volume are identified to obtain the spatial attention weights, including: Max pooling is performed on the first cost body to obtain the second local feature information; The first cost body is subjected to average pooling to obtain the second overall feature information; The second local feature information and the second global feature information are filtered along the spatial direction to obtain key spatial location information related to depth matching; The key spatial location information is activated to obtain the spatial attention weight.

8. The method for constructing a three-dimensional human body model according to claim 5, characterized in that, Based on the deep attention mechanism, a convolution operation is performed along the depth dimension of the second cost body to obtain deep attention weights, including: Max pooling is performed on the second cost body to obtain the third local feature information; Perform average pooling on the second cost body to obtain the third overall feature information; Based on a specified convolutional kernel, the third local feature information and the third global feature information are depth-enhanced along the depth direction, and the depth-enhanced information is activated to obtain the depth attention weights.

9. The method for constructing a three-dimensional human body model according to claim 1, characterized in that, Based on the parameterized model corresponding to the monocular human image, a priori 3D feature volume of the human skeleton is generated, including: Based on the human skeleton structure in the parameterized model, the three-dimensional spatial layout features of the human skeleton corresponding to the monocular human image are obtained. Obtain the two-dimensional projection coordinates of the three-dimensional spatial layout features; Based on the two-dimensional projection coordinates, extract the corresponding two-dimensional skeleton features from the monocular human image; The two-dimensional skeleton features and the three-dimensional spatial layout features are spliced ​​together to obtain the three-dimensional skeleton splicing features; The prior 3D feature body is obtained based on the acquired spatial query points and the 3D skeleton splicing features. The spatial query points are used to query relevant nodes in the 3D skeleton splicing features.

10. The method for constructing a three-dimensional human body model according to claim 9, characterized in that, The prior 3D feature body is obtained based on the acquired spatial query points and the splicing features of the 3D skeleton, including: Calculate the relative distance between the joints in the three-dimensional skeleton splicing features and the spatial query point; The correlation between the key points and the spatial query points is detected based on a threshold function. Based on the detection results, a feature representation of the human skeleton based on the spatial query points is obtained; The prior three-dimensional feature body is obtained based on the aforementioned feature representation.

11. The method for constructing a three-dimensional human body model according to any one of claims 1 to 10, characterized in that, Based on the normal map, the enhanced depth information, and the prior 3D feature volume of the human skeleton, the 3D human model is constructed, including: The normal map of the monocular human image includes the front normal map and the back normal map of the human body. The front normal map and the back normal map are used to provide information on the front and back surfaces of the human body. Based on the projection points in the prior three-dimensional feature volume, features are extracted from the front normal map, the back normal map and the monocular human image to obtain two-dimensional features. Based on the location points in the monocular human image, the two-dimensional features are sampled to obtain the sampled features; The sampled features, the enhanced depth information, and the prior 3D feature volume are fused using the features of the projection points to obtain fused features; The fusion features are used as additional parameters of the implicit function to solve the implicit function, thereby constructing the three-dimensional human body model.

12. A three-dimensional human body model construction device, characterized in that, include: The first generation module is used to generate a normal map of the monocular human image in response to the acquired monocular human image. A lightweight depth processing module is used to input the monocular human image into a lightweight depth model comprising a multi-scale feature extractor, a cost volume construction module, and a lightweight cost volume regularization network module, to construct an initial cost volume of the monocular human image based on the lightweight depth model, and to perform lightweight regularization processing on the initial cost volume to obtain enhanced depth information of the monocular human image. The module includes: inputting the monocular human image and a reference image into the multi-scale feature extractor and performing discrete wavelet transform to extract first image features at multiple depth levels of the monocular human image and second image features at multiple depth levels of the reference image; inputting the first image features and the second image features into the cost volume construction module to apply affine transform to the first image features based on the second image features at different viewpoints to construct the initial cost volume; and performing lightweight regularization processing on the initial cost volume based on the lightweight cost volume regularization network module to obtain the enhanced depth information. The second generation module is used to generate a priori three-dimensional feature volume of the human skeleton based on the parameterized model corresponding to the monocular human image. A construction module is used to construct the three-dimensional human model based on the normal map, the enhanced depth information, and the prior three-dimensional feature volume of the human skeleton.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the three-dimensional human body model construction method according to any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional human body model construction method according to any one of claims 1 to 11.

15. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional human body model construction method according to any one of claims 1 to 11.

Images