Human head image standardized embedding representation method, device, equipment and medium

Abstract

The invention discloses a human head image standardized embedding representation method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence in the fields of finance, medical treatment, insurance, banking and the like, and the method comprises the steps: constructing a three-dimensional standardized space, and defining a learnable potential feature grid in the three-dimensional standardized space; constructing and training an embedded network, and performing joint optimization training on the network and the feature grid based on a training data set comprising a plurality of head images and a ground true value point track pair generated by the training data set; and inputting a to-be-processed target head image into the trained embedded network to obtain a target coordinate graph which represents the three-dimensional coordinates of each pixel point in the image in the normalized space. According to the method, by establishing a shared normalized space and learning dense mapping from pixels to the space, a stable and robust pixel-level corresponding relation can be established for a complete head including hair and accessories, and the defects that a traditional method is incomplete in coverage, prone to being affected by shielding and sparse in correspondence are overcome.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for the standardized embedding representation of human head images. Background Technology

[0002] In fields such as finance, healthcare, insurance, and banking, where image presentation and identity verification are critical, high-fidelity human head modeling technology has significant application value. Currently, the mainstream technical solutions in this field mainly rely on two types of methods: one is tracking technology based on sparse keypoints, and the other is modeling methods based on parametric 3D deformation models (such as 3DMM).

[0003] Sparse keypoint tracking methods achieve coarse facial alignment and tracking by detecting and tracking a small number of semantically meaningful facial features, such as the corners of the eyes and mouth. However, the number of points processed by this method is extremely limited, making it impossible to cover the entire head area. In particular, it is completely ineffective in tracking and representing non-facial areas such as hair, glasses, earrings, hats, and clothing.

[0004] Parametric 3D modeling methods, by fitting a basic shape model obtained through statistical learning, can provide denser geometric information than sparse keypoints. However, their model structures are usually relatively simple, and the core modeling object is limited to the facial skin area, making it difficult to extend to parts that exceed the basic facial mesh topology, such as hair and complex headdresses. Therefore, they also cannot achieve modeling of the complete head.

[0005] Existing technologies have significant limitations. First, they lack completeness in coverage, focusing primarily on the face while neglecting non-facial regions that significantly contribute to the overall head shape. Second, they lack robustness in complex real-world scenarios; when the face is partially obscured by extreme poses, exaggerated expressions, or objects (such as hands or microphones), tracking and fitting methods relying on facial appearance information are prone to failure, leading to tracking loss or significant errors. Finally, there are bottlenecks in the quality of output correspondences; sparse keypoints cannot establish pixel-level dense correspondences, and the correspondences provided by parametric models in the facial region are limited by their mesh resolution and coverage, severely restricting the effectiveness of downstream applications such as high-precision texture mapping, subtle expression transfer, and high-quality geometric reconstruction. Summary of the Invention

[0006] This invention provides a method, apparatus, computer device, and storage medium for the normalized embedding representation of human head images. The technical problem it aims to solve is: how to provide an effective solution that can provide robust, stable, and dense pixel-level correspondences for a complete human head including hair and accessories.

[0007] In a first aspect, embodiments of the present invention provide a normalized embedding representation method for human head images, comprising: Construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space; Construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; The target head image to be processed is input into the trained embedding network to obtain a target coordinate map, which represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

[0008] Optionally, the joint optimization training of the embedding network and the latent feature grid based on training data includes: Obtain a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories contains a first pixel coordinate in the first training image and a second pixel coordinate in the second training image corresponding to the first pixel coordinate; The first training image and the second training image are respectively input into the embedding network to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. Based on the first coordinate map, obtain the first normalized coordinates corresponding to the coordinates of the first pixel; based on the second coordinate map, obtain the second normalized coordinates corresponding to the coordinates of the second pixel; through interpolation, obtain the first feature vector from the latent feature grid based on the first normalized coordinates, and obtain the second feature vector from the latent feature grid based on the second normalized coordinates; Based on the first feature vector and the second feature vector, a contrastive loss function is constructed, and the parameters of the embedded network and the values ​​of the latent feature grid are updated using the contrastive loss function to reduce the distance between the first feature vector and the second feature vector in the feature space.

[0009] Optionally, the joint optimization training of the embedding network and the latent feature grid based on the training data further includes: Using a pre-trained landmark detector, multiple first sparse keypoints are detected from the first training image, and multiple second sparse keypoints are detected from the second training image; For each first sparse keypoint in the first training image, the normalized coordinates corresponding to the first sparse keypoint are obtained from the first coordinate graph; for each second sparse keypoint in the second training image, the normalized coordinates corresponding to the second sparse keypoint are obtained from the second coordinate graph. Calculate the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first sparse keypoints and second sparse keypoints and the fixed target position coordinates that are pre-set in the normalized space and semantically bound to each keypoint. The landmark anchoring loss is combined with the contrastive loss function to update the parameters of the embedded network.

[0010] Optionally, the joint optimization training of the embedding network and the latent feature grid based on the training data further includes: A segmentation prediction head is added to the embedded network; For the first training image, the intermediate features extracted by the embedding network, or the features queried from the latent feature grid based on the first coordinate map, are input into the segmentation prediction head to obtain the predicted segmentation mask; Obtain the ground truth mask for the pseudo-segmentation of the head region preset for the first training image; Calculate the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; The segmentation loss is combined with the contrastive loss function and the landmark anchoring loss to update the parameters of the embedded network and the parameters of the segmentation prediction head.

[0011] Optionally, the step of obtaining a first feature vector from the latent feature grid based on the first normalized coordinates through interpolation, and obtaining a second feature vector from the latent feature grid based on the second normalized coordinates, includes: Determine the first minimum cube cell in the potential feature grid where the first normalized coordinates are located, and locate the eight first vertices of the first minimum cube cell; Obtain the eight first vertex feature vectors stored in the latent feature grid; Calculate the first weight of the spatial position of the first normalized coordinates relative to the eight first vertices of the first smallest cube unit; Based on the first weight of the spatial position, the feature vectors of the eight first vertices are weighted and summed to obtain the first feature vector; Determine the second minimum cube cell in the potential feature grid where the second normalized coordinates are located, and locate the eight second vertices of the second minimum cube cell; Obtain the eight second vertex feature vectors stored in the latent feature grid; Calculate the second weight of the spatial position of the second normalized coordinates relative to the eight second vertices of the second minimum cube element; Based on the second weight of the spatial position, the feature vectors of the eight second vertices are weighted and summed to obtain the second feature vector.

[0012] Optionally, the normalized space is a three-dimensional unit cube; after defining a learnable latent feature mesh in the normalized space, the method further includes: A three-dimensional Gaussian convolution kernel is applied to the feature vector of each grid point in the potential feature grid to filter it, so that the feature vectors of adjacent grid points in the potential feature grid transition smoothly in space.

[0013] Optionally, the method further includes: When the input is a single target head image, the target coordinate map is used to achieve pixel-level dense matching by performing nearest neighbor search between the coordinate maps corresponding to different images; or... The dense correspondences provided by the target coordinate map are combined with the fitting results of the preset parametric head model to provide an alignment basis for the texture mapping of the parametric head model; or, When the input is multiple target head images of the same head from different perspectives, the target coordinate maps corresponding to each image are combined, and 3D point clouds or surface meshes are reconstructed through triangulation or multi-view geometric constraint methods.

[0014] Secondly, embodiments of the present invention also provide a normalized embedding representation apparatus for human head images, which includes a unit for performing the above-described method.

[0015] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0017] This invention provides a method, apparatus, computer device, and storage medium for normalized embedding representation of human head images. The method includes: constructing a three-dimensional normalized space and defining a learnable latent feature grid within the normalized space; constructing and training an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; inputting the target head image to be processed into the trained embedding network to obtain a target coordinate map, wherein the target coordinate map represents the three-dimensional coordinates of each pixel in the target head image in the normalized space. By constructing a shared three-dimensional normalized space and training the embedding network to map image pixels to this space, the problem of dense correspondence of the entire head is effectively solved. First, the continuous three-dimensional volume characteristic preset in the normalized space allows the embedding network to map non-facial pixels such as hair and accessories, overcoming the limitation of traditional methods that only cover the face and achieving coverage of the entire head. Secondly, supervised training using field video point trajectories enables the network to learn geometric invariance under appearance changes, thus maintaining mapping stability even in the face of occlusion and extreme poses, significantly improving robustness. Finally, the network outputs the precise 3D coordinates of each pixel in the input image in the normalized space, forming a stable and continuous pixel-level dense correspondence, providing a high-quality foundation for downstream tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a normalized embedding representation method for human head images provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0025] Please see Figure 1 This invention provides a normalized embedding representation method for human head images, which includes the following steps: S1. Construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space.

[0026] In practice, the first step is to construct a three-dimensional normalized space. This space is typically defined as a normalized three-dimensional unit cube, with its three coordinate axes ranging from zero to one. This cube serves as an abstract, unified reference coordinate system independent of any specific head instance. Within this normalized space, a learnable latent feature grid needs to be defined. This latent feature grid is structurally a four-dimensional tensor. The first three dimensions correspond to the number of discretized grid points in the cube space along the three coordinate directions, while the fourth dimension represents the length of the feature vector stored at each grid point. For example, a grid can be defined with several grid points in each of the length, width, and height directions, with each grid point storing a feature vector of a specific dimension. These feature vectors will be used as model parameters for continuous optimization during subsequent training.

[0027] In some preferred embodiments, the normalized space is a three-dimensional unit cube; after defining a learnable latent feature grid in the normalized space, the method further includes: applying a three-dimensional Gaussian convolution kernel to filter the feature vector at each grid point in the latent feature grid, so that the feature vectors of adjacent grid points in the latent feature grid transition smoothly in space.

[0028] In practice, the normalized space is specifically defined as a three-dimensional unit cube, meaning that the values ​​of its three dimensions all range from zero to one, and its coordinates range from [0, 1]. This normalized cube serves as a standard container, simplifying the definition and calculation of coordinates.

[0029] Furthermore, after defining the learnable latent feature grid, a spatial smoothing step is added. This step involves applying a 3D Gaussian convolution kernel to the initialized latent feature grid for filtering. The size and standard deviation of the 3D Gaussian convolution kernel are preset hyperparameters that determine the strength of the smoothing. The filtering operation is performed in the three spatial dimensions of the feature grid. For each feature vector in the grid, the original feature value is replaced by a Gaussian weighted average of the features of its surrounding neighborhood grid points. It is important to note that filtering is performed independently on each feature channel, and the weights of the convolution kernel are fixed and do not participate in the learning process. This operation can be considered a fixed processing layer in the network, ensuring that the grid always maintains its spatial smoothness.

[0030] In this embodiment, a spatial smoothing step is introduced. By applying a smoothness prior constraint to the feature grid, a significant technical effect is achieved, improving the model's generalization ability, spatial continuity, and representation robustness. This effect stems from the explicit regularization of the implicit spatial structure of the feature grid. Without this constraint, the features of each grid point are learned independently, which may lead to huge differences in features between adjacent grid points, causing drastic jumps in the feature representation of neighboring regions within the normalized space. This discontinuity does not conform to the generally continuous and gradual spatial variation of head surface attributes (such as skin color and texture), easily leading to overfitting of the model to noise or random details in the training data, thus impairing its generalization ability.

[0031] After applying a 3D Gaussian filter, the features of each grid point are "smoothed" by the features of its neighborhood, tending to transition smoothly with the surrounding features. This brings multiple benefits. First, it forces the model to learn a spatially continuous and coherent feature field, which directly makes the coordinate map predicted by the embedded network smoother, reducing the generation of outlier prediction points and local geometric distortions, and improving the visual rationality of the output results. Furthermore, smoothness, as an effective regularization method, suppresses the model's memory of accidental and non-essential patterns in the training data, encouraging the model to capture more general and semantically relevant feature change patterns, thereby improving the model's robustness and generalization performance under unseen lighting, skin color, hairstyle, and other conditions. Finally, a smooth feature grid makes feature changes within the normalized space more regular and interpretable. Semantically similar regions form continuous subspaces with slowly changing feature values, facilitating semantic-based analysis and operations.

[0032] S2, Construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset.

[0033] In practice, an embedding network is constructed and trained. This embedding network is implemented as a deep convolutional neural network, taking a color image of a human head of arbitrary size as input and outputting a coordinate graph with the same height and width as the input image. Each pixel in this coordinate graph contains a three-dimensional coordinate value, representing the specific location of that pixel in the normalized space. The training process is a joint optimization process, optimizing all weight parameters of the embedding network and all feature vectors stored in the latent feature grid. The training data includes a large-scale training dataset of human head images and at least one set of ground truth point trajectory pairs generated from this dataset. These point trajectory pairs provide ground truth values ​​of pixel-level correspondences between different images, serving as supervision signals. Through training, the embedding network learns to map pixels corresponding to the same physical point on the head in different images to the same or very close coordinate positions in the normalized space.

[0034] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data includes: obtaining a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories includes the coordinates of a first pixel in the first training image and the coordinates of a second pixel in the second training image corresponding to the coordinates of the first pixel; inputting the first training image and the second training image into the embedding network respectively to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. The process involves: mapping the image; obtaining first normalized coordinates corresponding to the coordinates of the first pixel based on the first coordinate map; obtaining second normalized coordinates corresponding to the coordinates of the second pixel based on the second coordinate map; using interpolation, obtaining a first feature vector from the latent feature grid based on the first normalized coordinates, and obtaining a second feature vector from the latent feature grid based on the second normalized coordinates; constructing a contrastive loss function based on the first and second feature vectors, and using the contrastive loss function to update the parameters of the embedding network and the values ​​of the latent feature grid to reduce the distance between the first and second feature vectors in the feature space.

[0035] In practice, two images are first sampled from the training dataset, designated as the first training image and the second training image. These two images typically originate from different frames of the same video to ensure they depict the same person's head. Simultaneously, at least one pair of ground truth point trajectories associated with this pair of images is obtained from pre-generated trajectory data. This point trajectory pair is specifically represented as a two-dimensional coordinate pair, containing the position of a specific pixel in the first training image and the precisely matching pixel position in the second training image, representing the same physical point. In actual training, a large number of such point trajectory pairs are used to form a training batch to improve efficiency.

[0036] Furthermore, the first and second training images are input together into an embedding network with shared weights. The network processes the two images separately, outputting corresponding first and second coordinate maps. In the first coordinate map, the position corresponding to the first pixel coordinate in the point trajectory pair is found, thus obtaining the predicted first normalized coordinate of that point, which is a three-dimensional vector. Similarly, the second normalized coordinate corresponding to the second pixel coordinate is obtained in the second coordinate map.

[0037] Furthermore, based on the obtained first normalized coordinates, a query is performed within the defined learnable latent feature grid. Since normalized coordinates are continuous values, while the feature grid is discrete, an interpolation operation is needed to obtain the feature representation at that coordinate. Specifically, based on the value of the first normalized coordinates, an interpolation method is used to calculate a corresponding eigenvector from the feature grid, called the first eigenvector. The exact same operation is performed based on the second normalized coordinates to obtain the second eigenvector. This interpolation operation must be differentiable so that gradients can propagate.

[0038] Furthermore, based on the first and second feature vectors obtained from the query, a contrastive loss function is constructed. The core objective of this function is to minimize the distance between the feature vectors of known matching point pairs in the feature space, while maximizing the feature distance for mismatched point pairs. The gradient of this contrastive loss function with respect to the embedding network parameters and the feature values ​​in the latent feature grid is calculated using the backpropagation algorithm, and these parameters are updated using an optimization algorithm. By iterating this process repeatedly across the entire dataset, the embedding network and the feature grid are co-optimized, ultimately causing matching pixel pairs to converge in the normalized space and extracting similar features.

[0039] This embodiment effectively solves the core problems of unstable correspondence and weak generalization ability by directly utilizing pixel-level point trajectory ground truth as supervision and designing an optimization objective based on feature comparison. Its technical effectiveness stems primarily from the use of high-precision point trajectory pairs as strong supervision signals. These trajectory pairs originate from the motion paths of the same physical point in real video sequences, providing precise pixel correspondences across different poses and expressions. This constrains the network, forcing it to learn the invariant nature of head appearance variations—that is, regardless of head movement, pixels corresponding to the same physical point should be mapped to the same location in the normalized space—thus greatly improving the robustness and generalization ability of the learned representation.

[0040] Furthermore, by querying features from the latent feature grid and calculating the contrastive loss, this method creatively integrates coordinate regression with feature representation learning. The network not only learns to predict accurate coordinates, but the associated feature grid also learns to assign highly discriminative features to different regions in space. This makes the normalized space not only a geometric coordinate framework but also a feature field rich in semantic information. Driven by the contrastive loss, the feature vectors of matching point pairs move closer to each other in the feature space, forming tight clusters, while mismatched point pairs move further apart. This dual constraint mechanism ensures that the resulting dense correspondences possess both spatial continuity and strong feature discriminative power, providing accurate and information-rich intermediate representations for subsequent matching and reconstruction tasks. This fundamentally overcomes the shortcomings of sparse correspondences (insufficient information) and the ambiguity inherent in traditional methods in areas with weak texture.

[0041] In some preferred embodiments, the step of querying a first feature vector from the latent feature grid based on the first normalized coordinates and a second feature vector from the latent feature grid based on the second normalized coordinates via interpolation includes: determining the first minimum cube cell in the latent feature grid where the first normalized coordinates are located, and locating the eight first vertices of the first minimum cube cell; obtaining the eight first vertex feature vectors stored in the latent feature grid for the eight first vertices; calculating a first spatial position weight of the first normalized coordinates relative to the eight first vertices of the first minimum cube cell; performing a weighted summation of the eight first vertex feature vectors based on the first spatial position weight to obtain the first feature vector; determining the second minimum cube cell in the latent feature grid where the second normalized coordinates are located, and locating the eight second vertices of the second minimum cube cell; obtaining the eight second vertex feature vectors stored in the latent feature grid for the eight second vertices; calculating a second spatial position weight of the second normalized coordinates relative to the eight second vertices of the second minimum cube cell; and performing a weighted summation of the eight second vertex feature vectors based on the second spatial position weight to obtain the second feature vector.

[0042] In practice, for the first normalized coordinate, the specific smallest cubic unit in the discrete latent feature grid where the coordinate lies is first determined. This unit is uniquely determined by the integer coordinates of its lower left corner (i.e., the corner with the smallest index value in three dimensions) in the grid. This smallest cubic unit has eight vertices, namely the corner point and its three adjacent grid points in the three positive directions, which together constitute the eight corner points of a cube.

[0043] Furthermore, the feature vectors stored at each of the eight vertices are directly extracted from the latent feature grid. These eight feature vectors form the basis for interpolation.

[0044] Further, the spatial position weights of the first normalized coordinates relative to this smallest cubic unit are calculated. Specifically, the relative offset ratios of these coordinates within the unit along the three coordinate axes are calculated. These three ratios and their complements (one minus the ratio) together constitute the weighting coefficients used for interpolation.

[0045] Furthermore, based on these spatial location weights, the feature vectors of the eight vertices are weighted and summed. The weighted summation follows the trilinear interpolation rule: first, based on the offset ratio in one direction, two pairs of four vertices are interpolated to obtain two intermediate values; then, based on the offset ratio in the second direction, these two intermediate values ​​are interpolated to obtain a new intermediate value; finally, a final interpolation is performed based on the offset ratio in the third direction, ultimately obtaining the smooth and continuous first feature vector corresponding to the first normalized coordinates.

[0046] Repeating the exact same steps for the second normalized coordinates yields the second eigenvector, which will not be elaborated further in this invention.

[0047] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: using a pre-trained landmark detector to detect multiple first sparse keypoints from the first training image and multiple second sparse keypoints from the second training image; for each first sparse keypoint in the first training image, obtaining the normalized coordinates corresponding to the first sparse keypoint from the first coordinate map; for each second sparse keypoint in the second training image, obtaining the normalized coordinates corresponding to the second sparse keypoint from the second coordinate map; calculating the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first and second sparse keypoints and the fixed target position coordinates semantically bound to each keypoint in the pre-set normalized space; and combining the landmark anchoring loss with the contrastive loss function to update the parameters of the embedding network.

[0048] In practice, a landmark detector is first pre-trained. This landmark detector is a mature model capable of automatically locating sparse keypoints with clear semantics from face images, such as the corners of the eyes, the tip of the nose, and the corners of the mouth. During training, for the first and second training images in the current batch, this landmark detector is used for inference to obtain two sets of two-dimensional image coordinates of keypoints, and the two sets of keypoints have the same semantic meaning in the order of their indices.

[0049] Furthermore, it is necessary to obtain the predicted normalized coordinates of these keypoints in their corresponding coordinate graphs. For each keypoint in the first training image, by consulting the first coordinate graph generated by the embedding network, the first normalized coordinates predicted by the network at the keypoint's image coordinate location are directly read. The same operation is performed on the keypoints in the second training image to obtain the second normalized coordinates from the second coordinate graph.

[0050] Further, the landmark anchoring loss is calculated. Before training begins, a fixed target location coordinate is predefined for each semantic keypoint within the normalized space (i.e., the unit cube). For example, it is predefined that "nose tip" always corresponds to a specific 3D coordinate in space. The landmark anchoring loss is calculated by measuring the difference between the normalized coordinates of all predicted keypoints and their respective predefined fixed target coordinates. Typically, the sum of the squares or absolute values ​​of all coordinate differences is used as the loss value.

[0051] Furthermore, during the backpropagation process of training, the calculated landmark anchoring loss is combined with the aforementioned contrastive loss function. A common approach is to assign weights to the two losses and then sum them to obtain a total loss. This total loss is used to update the parameters of the embedding network. Through this joint optimization, the embedding network strives to meet the consistency requirement of point trajectories while also being forced to map all semantically specific keypoints in the images to that unique, fixed location in the normalized space.

[0052] This embodiment introduces a landmark anchoring step, using sparse but semantically clear keypoints as spatial "anchors" to inject a strong semantic structure prior into the entire learning system. This results in significant technical effects such as improved semantic consistency, elimination of spatial ambiguity, and enhanced model interpretability. The core of this effect lies in establishing an absolute semantic coordinate system, independent of specific image content, for the normally normalized space that might otherwise drift freely. Without this constraint, relying solely on point trajectory contrast loss, the network might learn a geometrically continuous but arbitrarily aligned space with uncertain origin, axis, and scale. Landmark anchoring loss enforces, for example, that the "outer corner of the left eye" in all images must be mapped to an absolute coordinate A in space. This ensures that the learned normalized space possesses a stable and interpretable semantic structure.

[0053] Furthermore, this semantic anchoring brings multiple benefits. First, it greatly enhances the semantic consistency of the learned representations. Regardless of changes in the identity, pose, or expression of the input face, the position of specific semantic parts in the normalized space remains constant. This ensures the direct comparability of features extracted from the same semantic region from different images, greatly facilitating tasks such as semantic-based image editing and attribute transfer. Furthermore, it effectively eliminates the ambiguity of the normalized space. Contrast loss primarily constrains the relative positional relationships of matching point pairs, but its constraints on their absolute positions are weak. Landmark loss provides strong absolute positional constraints, acting as a "calibrator" to ensure that the mapping position and orientation of the entire head in space are generally fixed and reasonable, preventing the network from learning invalid solutions that satisfy the contrast loss but result in a distorted overall structure, thus stabilizing the training process and improving model performance. Furthermore, this makes the normalized space itself a highly interpretable and interactive tool. Users can explicitly know that a specific region in the space corresponds to the "chin" or "forehead" of all head images. This interpretability not only facilitates understanding and debugging the model, but also lays a solid foundation for developing more advanced, semantically guided visual applications.

[0054] It should be noted that a landmark detector is a pre-trained model capable of automatically locating sparse keypoints with clear anatomical or semantic significance in facial images. These keypoints typically include the corners of the eyes, the tip of the nose, the corners of the mouth, and the outline of the eyebrows. Its function is to take a facial image as input and output a series of two-dimensional pixel coordinates that correspond one-to-one with specific semantic parts in the image.

[0055] The training process of this landmark detector is independent of the main workflow of this invention. Its training is typically based on a supervised learning paradigm, utilizing a large-scale dataset of labeled facial keypoints. Specifically, a deep convolutional neural network is constructed as the main body of the detector. Its training objective is to minimize the difference between the network-predicted keypoint coordinates and the manually labeled true coordinates in the dataset. This difference is quantified using a coordinate regression loss function (such as mean squared error loss). The network parameters are iteratively optimized through backpropagation, ultimately enabling the network to accurately regress sparse keypoint coordinates from the input image. The trained detector, as a fixed module, is used in this invention to provide sparse semantic constraint signals.

[0056] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: adding a segmentation prediction head to the embedding network; inputting the intermediate features extracted by the embedding network, or features queried from the latent feature grid based on the first coordinate map, into the segmentation prediction head for the first training image to obtain a predicted segmentation mask; obtaining a pseudo-segmentation ground truth mask for the head region preset for the first training image; calculating the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; and combining the segmentation loss with the contrast loss function and the landmark anchoring loss to update the parameters of the embedding network and the parameters of the segmentation prediction head.

[0057] In practice, the first step is to extend the structure of the embedded network by adding a segmentation prediction head at an appropriate location in its network architecture. This segmentation prediction head is typically a lightweight convolutional module whose input is a feature map and whose output is a multi-channel segmentation score map of the same size as the input image, with each channel corresponding to a semantic category.

[0058] During training iterations, for the current first training image, in addition to performing coordinate prediction and feature lookup processes, segmentation prediction is also required. There are two specific paths: one is to directly input the feature map extracted from an intermediate layer of the embedded network into the segmentation prediction head; the other is to first use the first coordinate map output by the network to perform a dense lookup of the latent feature grid, obtaining a feature map where each pixel has a feature vector, and then input this feature map into the segmentation prediction head. The segmentation prediction head ultimately outputs a predicted segmentation mask.

[0059] Furthermore, a pseudo-ground truth mask for supervision needs to be prepared for the first training image. This mask does not require manual annotation and can be used as a pseudo-ground truth by reasoning on the first training image through a general semantic segmentation model pre-trained on a large dataset.

[0060] Further, the segmentation loss is calculated. This loss measures the difference between the predicted segmentation mask and the pseudo-segmentation ground truth mask, and is typically calculated using the pixel-wise cross-entropy loss function.

[0061] Furthermore, the calculated segmentation loss is combined with the previously obtained contrast loss and landmark anchoring loss. A weighted summation method is typically used to construct the total loss function. During backpropagation, this total loss is simultaneously used to update the backbone parameters of the embedding network, the segmentation prediction head parameters, and indirectly optimize the latent feature grid through the coordinate graph. Through this multi-task joint training, the embedding network is motivated to learn feature representations that not only benefit coordinate regression and feature matching but also clearly distinguish different semantic regions of the head.

[0062] The segmentation-assisted training step in this embodiment introduces pixel-level region semantic supervision, forcing the network to extract more discriminative features through a multi-task learning paradigm. This enhances the semantic consistency of the representation and its ability to handle complex structures at a deeper level. The technical effect stems primarily from the semantic constraints provided by the segmentation loss at the continuous region level. Point trajectory loss focuses on the correspondence of "points," landmark loss focuses on the location of specific "points," while segmentation loss focuses on the correct classification of the entire "surface" or "region." This supervision forces the network to distinguish the fundamental differences between different semantic regions, such as hair and skin, or eyes and cheeks, during feature learning. Therefore, features corresponding to different semantic regions in the latent feature grid naturally separate, enhancing the semantic discriminative power of the features.

[0063] Furthermore, this enhanced semantic discriminative power directly improves the reliability of dense correspondences in regions with blurred or weak textures. For example, in large areas of smooth skin on the face, there is a lack of significant texture features for low-level matching, making traditional methods prone to failure. However, segmentation tasks require the network to correctly identify this region as "skin." To meet this requirement, the network learns high-level features with semantic commonalities to "skin" for this region. This high-level semantic consistency can supplement low-level texture matching, guiding the network to generate reasonable and semantically consistent coordinate predictions even in smooth regions, thereby improving the robustness and reasonableness of the overall correspondences.

[0064] Furthermore, multi-task learning itself serves as an effective regularization method, enabling the network to learn more general and applicable feature representations. The network cannot simply overfit to pixel matching; it must also consider keypoint localization and region segmentation. This shared representation learning process drives the network to uncover more fundamental structural and semantic information in images, thus exhibiting stronger generalization performance when faced with novel hairstyles, accessories, or extreme poses. This step solidifies the overall technical solution's effectiveness from a semantic perspective.

[0065] S3, Input the target head image to be processed into the trained embedding network to obtain a target coordinate map, wherein the target coordinate map represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

[0066] In practice, during the inference application phase, any new target head image to be processed is input into the already trained embedding network. After forward propagation computation, the network directly outputs a target coordinate map. This coordinate map records the three-dimensional coordinate values ​​in normalized space corresponding to each pixel in the input image in matrix form, thus completing the dense mapping from a two-dimensional image to a normalized three-dimensional space.

[0067] This invention systematically improves the integrity, robustness, and correspondence quality of head modeling by establishing a shared, structured, normalized three-dimensional space and training a network to map image pixels to this space.

[0068] Firstly, this invention achieves full coverage of the head. Since the normalized space is a pre-defined, continuous three-dimensional volume, the embedded network learns to project all pixels of the image, including facial skin, hair, ears, neck, and various accessories, to their corresponding positions within this space. This fundamentally breaks through the constraints of traditional parametric models limited by a fixed facial mesh topology, making it possible to model non-rigid, non-facial components such as hairstyles and glasses, thus solving the problem of incompleteness in traditional methods.

[0069] Furthermore, this invention significantly enhances stability in complex real-world scenarios. The supervisory signals relied upon for network training originate from real point trajectory pairs between images, which can be automatically acquired from diverse videos containing various poses, expressions, lighting conditions, and partial occlusions. Through learning and optimization, the embedded network can capture the inherent geometric consistency under drastic changes in head appearance, thus stably mapping visible portions and even reasonably inferred occluded portions to the correct region in the normalized space even when faced with large-angle rotations, partial occlusion, or the wearing of accessories. This overcomes the shortcomings of traditional trackers, such as easy failure and poor robustness under occlusion.

[0070] Furthermore, this invention generates stable and dense pixel-level correspondences. The target coordinate map output by the network assigns a precise coordinate in a normalized space to each pixel of the input image, forming a dense and continuous correspondence representation. Compared to the limited correspondences provided by sparse keypoints, this comprehensive dense correspondence provides a crucial data foundation for downstream high-fidelity texture mapping, nuanced facial expression and pose transfer, and high-quality 3D reconstruction tasks, making it possible to achieve unprecedented visual detail and geometric coherence.

[0071] Furthermore, in some preferred embodiments, the method further includes: when the input is a single target head image, using the target coordinate map, performing nearest neighbor search between coordinate maps corresponding to different images to achieve pixel-level dense matching; or, combining the dense correspondence provided by the target coordinate map with the fitting result of a preset parameterized head model to provide an alignment basis for the texture mapping of the parameterized head model; or, when the input is multiple target head images of the same head from different perspectives, combining the target coordinate maps corresponding to each image, and using triangulation or multi-view geometric constraint methods to reconstruct three-dimensional point clouds or surface meshes.

[0072] In practice, the applications based on the obtained target coordinate map fall into three categories, as follows: Firstly, the first type involves achieving pixel-level dense matching. When given two head images and their corresponding target coordinate maps, to find the point in one image that best matches a specific pixel in the other image, a nearest neighbor search can be performed in the target coordinate map. This involves calculating the pixel's coordinates in the normalized space and then finding the pixel in the other image's coordinate map with the closest Euclidean distance to it, identifying it as the matching point. This process can efficiently achieve a global dense correspondence between two images.

[0073] Furthermore, the second category involves texture alignment for auxiliary parametric head models. In this application, a parametric 3D head mesh model is first obtained by fitting a single image using traditional methods. Simultaneously, the target coordinate map of this image is obtained using this invention. The key step is establishing a mapping relationship between normalized spatial coordinates and points on the 3D mesh surface. Using this mapping relationship, the pixel colors of the input image can be directly assigned to the corresponding positions on the 3D mesh model surface based on the precise correspondence provided by the target coordinate map, thereby achieving high-precision texture mapping and overcoming the texture blurring problem caused by sparse correspondence in traditional methods.

[0074] Furthermore, the third category is multi-view 3D reconstruction. When multiple images of the same head from different perspectives and their corresponding target coordinate maps are available, the normalized space itself can be considered as the reconstructed 3D space. The basic principle is that pixels from different perspectives that correspond to the same 3D point in the real world should be mapped to the same or very close coordinates in the normalized space. By integrating mapping evidence of the same spatial location from multiple coordinate maps and utilizing multi-view geometric constraints (such as triangulation), an accurate 3D point cloud can be optimized and calculated, thereby reconstructing the complete 3D surface geometry of the head.

[0075] The powerful practical value and generalization ability of the normalized embedding representation generated by this invention directly transform dense, robust, and semantically consistent coordinate information into an effective tool for solving multiple core visual tasks. In dense matching, it achieves high-precision pixel alignment covering the entire head, insensitive to lighting and facial expression changes. Traditional matching methods based on appearance features fail in areas with weak texture, while this invention utilizes normalized spatial coordinates as the matching basis, fundamentally eliminating dependence on underlying textures and achieving stable and reliable global dense correspondence. In texture alignment, it greatly improves the realism and detail fidelity of parametric model texture maps. Through precise mapping established by the target coordinate map, accurate positioning from the surface of the 3D model to the pixels of the 2D image is achieved, enabling the rendering result to perfectly preserve all high-frequency details such as skin texture and hair in the input image. In multi-view reconstruction, it provides a novel and efficient learning-based reconstruction paradigm. It pre-"upgrades" 2D image information to a shared normalized 3D space, simplifying the complex matching and optimization problems in traditional multi-view stereo vision. Since embedded networks are already robust to single-image occlusion, this method can effectively fill in the missing parts by utilizing information from the unoccluded viewpoint, thereby improving the integrity and robustness of 3D reconstruction.

[0076] Corresponding to the above-described method for normalized embedding representation of human head images, the present invention also provides a device for normalized embedding representation of human head images. This device includes a unit for executing the above-described method for normalized embedding representation of human head images, and can be configured in a terminal or server. Specifically, the device includes: A building unit is used to construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space; A training unit is used to construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; The input unit is used to input the target head image to be processed into the trained embedding network to obtain a target coordinate map, wherein the target coordinate map represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

[0077] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data includes: Obtain a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories contains a first pixel coordinate in the first training image and a second pixel coordinate in the second training image corresponding to the first pixel coordinate; The first training image and the second training image are respectively input into the embedding network to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. Based on the first coordinate map, obtain the first normalized coordinates corresponding to the coordinates of the first pixel; based on the second coordinate map, obtain the second normalized coordinates corresponding to the coordinates of the second pixel; through interpolation, obtain the first feature vector from the latent feature grid based on the first normalized coordinates, and obtain the second feature vector from the latent feature grid based on the second normalized coordinates; Based on the first feature vector and the second feature vector, a contrastive loss function is constructed, and the parameters of the embedded network and the values ​​of the latent feature grid are updated using the contrastive loss function to reduce the distance between the first feature vector and the second feature vector in the feature space.

[0078] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: Using a pre-trained landmark detector, multiple first sparse keypoints are detected from the first training image, and multiple second sparse keypoints are detected from the second training image; For each first sparse keypoint in the first training image, the normalized coordinates corresponding to the first sparse keypoint are obtained from the first coordinate graph; for each second sparse keypoint in the second training image, the normalized coordinates corresponding to the second sparse keypoint are obtained from the second coordinate graph. Calculate the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first sparse keypoints and second sparse keypoints and the fixed target position coordinates that are pre-set in the normalized space and semantically bound to each keypoint. The landmark anchoring loss is combined with the contrastive loss function to update the parameters of the embedded network.

[0079] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: A segmentation prediction head is added to the embedded network; For the first training image, the intermediate features extracted by the embedding network, or the features queried from the latent feature grid based on the first coordinate map, are input into the segmentation prediction head to obtain the predicted segmentation mask; Obtain the ground truth mask for the pseudo-segmentation of the head region preset for the first training image; Calculate the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; The segmentation loss is combined with the contrastive loss function and the landmark anchoring loss to update the parameters of the embedded network and the parameters of the segmentation prediction head.

[0080] In some preferred embodiments, the step of querying the first feature vector from the latent feature grid based on the first normalized coordinates through interpolation, and querying the second feature vector from the latent feature grid based on the second normalized coordinates, includes: Determine the first minimum cube cell in the potential feature grid where the first normalized coordinates are located, and locate the eight first vertices of the first minimum cube cell; Obtain the eight first vertex feature vectors stored in the latent feature grid; Calculate the first weight of the spatial position of the first normalized coordinates relative to the eight first vertices of the first smallest cube unit; Based on the first weight of the spatial position, the feature vectors of the eight first vertices are weighted and summed to obtain the first feature vector; Determine the second minimum cube cell in the potential feature grid where the second normalized coordinates are located, and locate the eight second vertices of the second minimum cube cell; Obtain the eight second vertex feature vectors stored in the latent feature grid; Calculate the second weight of the spatial position of the second normalized coordinates relative to the eight second vertices of the second minimum cube element; Based on the second weight of the spatial position, the feature vectors of the eight second vertices are weighted and summed to obtain the second feature vector.

[0081] In some preferred embodiments, the normalized space is a three-dimensional unit cube; after defining a learnable latent feature mesh in the normalized space, the method further includes: A three-dimensional Gaussian convolution kernel is applied to the feature vector of each grid point in the potential feature grid to filter it, so that the feature vectors of adjacent grid points in the potential feature grid transition smoothly in space.

[0082] In some preferred embodiments, it further includes: The application unit is used to, when the input is a single target head image, utilize the target coordinate map to achieve pixel-level dense matching by performing nearest neighbor search between coordinate maps corresponding to different images; or, The dense correspondences provided by the target coordinate map are combined with the fitting results of the preset parametric head model to provide an alignment basis for the texture mapping of the parametric head model; or, When the input is multiple target head images of the same head from different perspectives, the target coordinate maps corresponding to each image are combined, and 3D point clouds or surface meshes are reconstructed through triangulation or multi-view geometric constraint methods.

[0083] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned human head image normalization embedding representation device and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0084] The aforementioned human head image normalization and embedding representation device can be implemented as a computer program, which can, for example... Figure 2 It runs on the computer device shown.

[0085] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0086] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0087] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a normalized embedded representation method for a human head image.

[0088] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0089] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a normalized embedding representation method for human head images.

[0090] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0091] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: Construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space; Construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; The target head image to be processed is input into the trained embedding network to obtain a target coordinate map, which represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

[0092] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data includes: Obtain a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories contains a first pixel coordinate in the first training image and a second pixel coordinate in the second training image corresponding to the first pixel coordinate; The first training image and the second training image are respectively input into the embedding network to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. Based on the first coordinate map, obtain the first normalized coordinates corresponding to the coordinates of the first pixel; based on the second coordinate map, obtain the second normalized coordinates corresponding to the coordinates of the second pixel; through interpolation, obtain the first feature vector from the latent feature grid based on the first normalized coordinates, and obtain the second feature vector from the latent feature grid based on the second normalized coordinates; Based on the first feature vector and the second feature vector, a contrastive loss function is constructed, and the parameters of the embedded network and the values ​​of the latent feature grid are updated using the contrastive loss function to reduce the distance between the first feature vector and the second feature vector in the feature space.

[0093] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: Using a pre-trained landmark detector, multiple first sparse keypoints are detected from the first training image, and multiple second sparse keypoints are detected from the second training image; For each first sparse keypoint in the first training image, the normalized coordinates corresponding to the first sparse keypoint are obtained from the first coordinate graph; for each second sparse keypoint in the second training image, the normalized coordinates corresponding to the second sparse keypoint are obtained from the second coordinate graph. Calculate the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first sparse keypoints and second sparse keypoints and the fixed target position coordinates that are pre-set in the normalized space and semantically bound to each keypoint. The landmark anchoring loss is combined with the contrastive loss function to update the parameters of the embedded network.

[0094] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: A segmentation prediction head is added to the embedded network; For the first training image, the intermediate features extracted by the embedding network, or the features queried from the latent feature grid based on the first coordinate map, are input into the segmentation prediction head to obtain the predicted segmentation mask; Obtain the ground truth mask for the pseudo-segmentation of the head region preset for the first training image; Calculate the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; The segmentation loss is combined with the contrastive loss function and the landmark anchoring loss to update the parameters of the embedded network and the parameters of the segmentation prediction head.

[0095] In some preferred embodiments, the step of querying the first feature vector from the latent feature grid based on the first normalized coordinates through interpolation, and querying the second feature vector from the latent feature grid based on the second normalized coordinates, includes: Determine the first minimum cube cell in the potential feature grid where the first normalized coordinates are located, and locate the eight first vertices of the first minimum cube cell; Obtain the eight first vertex feature vectors stored in the latent feature grid; Calculate the first weight of the spatial position of the first normalized coordinates relative to the eight first vertices of the first smallest cube unit; Based on the first weight of the spatial position, the feature vectors of the eight first vertices are weighted and summed to obtain the first feature vector; Determine the second minimum cube cell in the potential feature grid where the second normalized coordinates are located, and locate the eight second vertices of the second minimum cube cell; Obtain the eight second vertex feature vectors stored in the latent feature grid; Calculate the second weight of the spatial position of the second normalized coordinates relative to the eight second vertices of the second minimum cube element; Based on the second weight of the spatial position, the feature vectors of the eight second vertices are weighted and summed to obtain the second feature vector.

[0096] In some preferred embodiments, the normalized space is a three-dimensional unit cube; after defining a learnable latent feature mesh in the normalized space, the method further includes: A three-dimensional Gaussian convolution kernel is applied to the feature vector of each grid point in the potential feature grid to filter it, so that the feature vectors of adjacent grid points in the potential feature grid transition smoothly in space.

[0097] In some preferred embodiments, the method further includes: When the input is a single target head image, the target coordinate map is used to achieve pixel-level dense matching by performing nearest neighbor search between the coordinate maps corresponding to different images; or... The dense correspondences provided by the target coordinate map are combined with the fitting results of the preset parametric head model to provide an alignment basis for the texture mapping of the parametric head model; or, When the input is multiple target head images of the same head from different perspectives, the target coordinate maps corresponding to each image are combined, and 3D point clouds or surface meshes are reconstructed through triangulation or multi-view geometric constraint methods.

[0098] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0099] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0100] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the following steps: Construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space; Construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; The target head image to be processed is input into the trained embedding network to obtain a target coordinate map, which represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

[0101] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data includes: Obtain a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories contains a first pixel coordinate in the first training image and a second pixel coordinate in the second training image corresponding to the first pixel coordinate; The first training image and the second training image are respectively input into the embedding network to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. Based on the first coordinate map, obtain the first normalized coordinates corresponding to the coordinates of the first pixel; based on the second coordinate map, obtain the second normalized coordinates corresponding to the coordinates of the second pixel; through interpolation, obtain the first feature vector from the latent feature grid based on the first normalized coordinates, and obtain the second feature vector from the latent feature grid based on the second normalized coordinates; Based on the first feature vector and the second feature vector, a contrastive loss function is constructed, and the parameters of the embedded network and the values ​​of the latent feature grid are updated using the contrastive loss function to reduce the distance between the first feature vector and the second feature vector in the feature space.

[0102] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: Using a pre-trained landmark detector, multiple first sparse keypoints are detected from the first training image, and multiple second sparse keypoints are detected from the second training image; For each first sparse keypoint in the first training image, the normalized coordinates corresponding to the first sparse keypoint are obtained from the first coordinate graph; for each second sparse keypoint in the second training image, the normalized coordinates corresponding to the second sparse keypoint are obtained from the second coordinate graph. Calculate the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first sparse keypoints and second sparse keypoints and the fixed target position coordinates that are pre-set in the normalized space and semantically bound to each keypoint. The landmark anchoring loss is combined with the contrastive loss function to update the parameters of the embedded network.

[0103] In some preferred embodiments, the joint optimization training of the embedding network and the latent feature grid based on training data further includes: A segmentation prediction head is added to the embedded network; For the first training image, the intermediate features extracted by the embedding network, or the features queried from the latent feature grid based on the first coordinate map, are input into the segmentation prediction head to obtain the predicted segmentation mask; Obtain the ground truth mask for the pseudo-segmentation of the head region preset for the first training image; Calculate the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; The segmentation loss is combined with the contrastive loss function and the landmark anchoring loss to update the parameters of the embedded network and the parameters of the segmentation prediction head.

[0104] In some preferred embodiments, the step of querying the first feature vector from the latent feature grid based on the first normalized coordinates through interpolation, and querying the second feature vector from the latent feature grid based on the second normalized coordinates, includes: Determine the first minimum cube cell in the potential feature grid where the first normalized coordinates are located, and locate the eight first vertices of the first minimum cube cell; Obtain the eight first vertex feature vectors stored in the latent feature grid; Calculate the first weight of the spatial position of the first normalized coordinates relative to the eight first vertices of the first smallest cube unit; Based on the first weight of the spatial position, the feature vectors of the eight first vertices are weighted and summed to obtain the first feature vector; Determine the second minimum cube cell in the potential feature grid where the second normalized coordinates are located, and locate the eight second vertices of the second minimum cube cell; Obtain the eight second vertex feature vectors stored in the latent feature grid; Calculate the second weight of the spatial position of the second normalized coordinates relative to the eight second vertices of the second minimum cube element; Based on the second weight of the spatial position, the feature vectors of the eight second vertices are weighted and summed to obtain the second feature vector.

[0105] In some preferred embodiments, the normalized space is a three-dimensional unit cube; after defining a learnable latent feature mesh in the normalized space, the method further includes: A three-dimensional Gaussian convolution kernel is applied to the feature vector of each grid point in the potential feature grid to filter it, so that the feature vectors of adjacent grid points in the potential feature grid transition smoothly in space.

[0106] In some preferred embodiments, the method further includes: When the input is a single target head image, the target coordinate map is used to achieve pixel-level dense matching by performing nearest neighbor search between the coordinate maps corresponding to different images; or... The dense correspondences provided by the target coordinate map are combined with the fitting results of the preset parametric head model to provide an alignment basis for the texture mapping of the parametric head model; or, When the input is multiple target head images of the same head from different perspectives, the target coordinate maps corresponding to each image are combined, and 3D point clouds or surface meshes are reconstructed through triangulation or multi-view geometric constraint methods.

[0107] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0109] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0110] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0114] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for normalized embedding representation of human head images, characterized in that, include: Construct a three-dimensional normalized space and define a learnable latent feature mesh in the normalized space; Construct and train an embedding network, wherein the training process includes: jointly optimizing and training the embedding network and the latent feature grid based on training data, wherein the training data includes: a training dataset containing multiple human head images, and at least one set of ground truth point trajectory pairs generated from the training dataset; The target head image to be processed is input into the trained embedding network to obtain a target coordinate map, which represents the three-dimensional coordinates of each pixel in the target head image in the normalized space.

2. The method for normalized embedding representation of human head images according to claim 1, characterized in that, The joint optimization training of the embedding network and the latent feature grid based on training data includes: Obtain a first training image and a second training image from the training dataset, and at least one pair of ground truth point trajectories between the first training image and the second training image, wherein each pair of point trajectories contains a first pixel coordinate in the first training image and a second pixel coordinate in the second training image corresponding to the first pixel coordinate; The first training image and the second training image are respectively input into the embedding network to obtain a first coordinate map corresponding to the first training image and a second coordinate map corresponding to the second training image. Based on the first coordinate map, obtain the first normalized coordinates corresponding to the coordinates of the first pixel; based on the second coordinate map, obtain the second normalized coordinates corresponding to the coordinates of the second pixel; through interpolation, obtain the first feature vector from the latent feature grid based on the first normalized coordinates, and obtain the second feature vector from the latent feature grid based on the second normalized coordinates; Based on the first feature vector and the second feature vector, a contrastive loss function is constructed, and the parameters of the embedded network and the values ​​of the latent feature grid are updated using the contrastive loss function to reduce the distance between the first feature vector and the second feature vector in the feature space.

3. The method for normalized embedding representation of human head images according to claim 2, characterized in that, The joint optimization training of the embedded network and the latent feature grid based on training data further includes: Using a pre-trained landmark detector, multiple first sparse keypoints are detected from the first training image, and multiple second sparse keypoints are detected from the second training image; For each first sparse keypoint in the first training image, the normalized coordinates corresponding to the first sparse keypoint are obtained from the first coordinate graph; for each second sparse keypoint in the second training image, the normalized coordinates corresponding to the second sparse keypoint are obtained from the second coordinate graph. Calculate the landmark anchoring loss, which is a measure of the difference between the normalized coordinates of all first sparse keypoints and second sparse keypoints and the fixed target position coordinates that are pre-set in the normalized space and semantically bound to each keypoint. The landmark anchoring loss is combined with the contrastive loss function to update the parameters of the embedded network.

4. The method for normalized embedding representation of human head images according to claim 3, characterized in that, The joint optimization training of the embedded network and the latent feature grid based on training data further includes: A segmentation prediction head is added to the embedded network; For the first training image, the intermediate features extracted by the embedding network, or the features queried from the latent feature grid based on the first coordinate map, are input into the segmentation prediction head to obtain the predicted segmentation mask; Obtain the ground truth mask for the pseudo-segmentation of the head region preset for the first training image; Calculate the segmentation loss between the predicted segmentation mask and the pseudo-segmentation ground truth mask; The segmentation loss is combined with the contrastive loss function and the landmark anchoring loss to update the parameters of the embedded network and the parameters of the segmentation prediction head.

5. The method for normalized embedding representation of human head images according to claim 2, characterized in that, The step of obtaining a first feature vector from the latent feature grid based on the first normalized coordinates through interpolation, and obtaining a second feature vector from the latent feature grid based on the second normalized coordinates, includes: Determine the first minimum cube cell in the potential feature grid where the first normalized coordinates are located, and locate the eight first vertices of the first minimum cube cell; Obtain the eight first vertex feature vectors stored in the latent feature grid; Calculate the first weight of the spatial position of the first normalized coordinates relative to the eight first vertices of the first smallest cube unit; Based on the first weight of the spatial position, the feature vectors of the eight first vertices are weighted and summed to obtain the first feature vector; Determine the second minimum cube cell in the potential feature grid where the second normalized coordinates are located, and locate the eight second vertices of the second minimum cube cell; Obtain the eight second vertex feature vectors stored in the latent feature grid; Calculate the second weight of the spatial position of the second normalized coordinates relative to the eight second vertices of the second minimum cube element; Based on the second weight of the spatial position, the feature vectors of the eight second vertices are weighted and summed to obtain the second feature vector.

6. The method for normalized embedding representation of human head images according to claim 1, characterized in that, The normalized space is a three-dimensional unit cube; after defining a learnable latent feature mesh in the normalized space, the method further includes: A three-dimensional Gaussian convolution kernel is applied to the feature vector of each grid point in the potential feature grid to filter it, so that the feature vectors of adjacent grid points in the potential feature grid transition smoothly in space.

7. The method for normalized embedding representation of human head images according to claim 1, characterized in that, The method further includes: When the input is a single target head image, the target coordinate map is used to achieve pixel-level dense matching by performing nearest neighbor search between the coordinate maps corresponding to different images; or... The dense correspondences provided by the target coordinate map are combined with the fitting results of the preset parametric head model to provide an alignment basis for the texture mapping of the parametric head model; or, When the input is multiple target head images of the same head from different perspectives, the target coordinate maps corresponding to each image are combined, and 3D point clouds or surface meshes are reconstructed through triangulation or multi-view geometric constraint methods.

8. A device for normalizing and embedding human head images, characterized in that, Includes a unit for performing the method as described in any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images