Method, device and equipment for removing shielding of real-time mixed reality head-mounted display

Abstract

The embodiment of the invention discloses a method, a device and equipment for removing shielding of a real-time mixed reality head-mounted display. One specific embodiment of the method comprises the following steps: generating implicit encoding information through an encoder of a model; through a decoder of the model, the following steps are executed: determining lower half face deformation information according to third preset feature constraint information, implicit coding information and lower half face geometric information; generating a spatial feature point information group according to the mapping association weight information, the first preset feature constraint information and the spatial feature point information set; generating upper half face deformation information according to the implicit coding information, the target weight, the upper half face key geometric information and the upper half face geometric information; changing the preset three-dimensional face geometric information to obtain changed three-dimensional face geometric information; and rendering the changed three-dimensional face geometric information to obtain shielding-free face image information. According to the embodiment, the accuracy and the stability of shielding removal are improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the fields of computer graphics and augmented reality, and more specifically to methods, apparatus, and devices for de-occlusion of real-time mixed reality head-mounted displays. Background Technology

[0002] In XR scene interaction, when a user wearing a head-mounted display interacts with an interactive object, both parties can use the head-mounted display to de-occlude the facial image of the interactive object, restoring the occluded upper half of the interactive object's face, thereby enabling the user to perceive the complete facial information of the interactive object in real time. Currently, the common methods for de-occluding facial images of interactive objects are image generation-based or geometry reconstruction-based. Image generation-based methods first use a generative adversarial network (GAN) to directly synthesize facial images or videos of the occluded area, and then use an encoder-decoder architecture to extract features of the visible area and generate the user's complete face. Geometry reconstruction-based methods reconstruct the 3D geometry of the face using devices such as 3D scanners to de-occlude the facial image of the user wearing the head-mounted display.

[0003] However, in practice, it has been found that when using the above method to de-occlude the image of a user wearing a head-mounted display device, the following technical problems often arise:

[0004] When using an encoder-decoder architecture to extract visible region features to generate facial images, the lack of consistency between frames leads to poor continuity and stability of facial dynamic changes, resulting in lower accuracy in occlusion removal and higher processing time for single-frame images. This results in poor real-time performance of facial image dynamic changes, leading to a poor user experience. Furthermore, occlusion removal using only two-dimensional features results in a limited field of view for capturing facial features from users wearing head-mounted displays, making it impossible to meet the requirements of binocular rendering and resulting in poor rendering effects. When using geometric reconstruction-based occlusion removal methods, the reliance on 3D scanners leads to high computational complexity, resulting in high computational resource consumption and long processing time for occlusion removal.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide methods, apparatuses, electronic devices, and computer-readable media for de-occlusion of real-time mixed reality head-mounted displays to address one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a real-time mixed reality head-mounted display occlusion removal method, the method comprising: generating implicit encoded information based on a preset facial image information group using an encoder included in a pre-trained occlusion removal 3D facial information generation model; and performing the following steps using a decoder included in the pre-trained occlusion removal 3D facial information generation model: determining lower half-face deformation information corresponding to the lower half-face geometry information based on a third preset feature constraint information, the aforementioned implicit encoded information, and lower half-face geometry information included in preset 3D facial geometry information; and determining lower half-face deformation information corresponding to the aforementioned lower half-face geometry information based on mapping association weight information, a first preset feature constraint information, and the space corresponding to the aforementioned preset 3D facial geometry information. The spatial feature point information set is used to generate a spatial feature point information group corresponding to the aforementioned preset three-dimensional facial geometry information. Based on the aforementioned implicit encoding information, target weight, key geometric information of the upper half of the face included in the aforementioned spatial feature point information group, and upper half of the face geometry included in the aforementioned preset three-dimensional facial geometry information, upper half of the face deformation information corresponding to the aforementioned upper half of the face geometry information is generated. Based on the aforementioned upper half of the face deformation information and the aforementioned lower half of the face deformation information, the aforementioned preset three-dimensional facial geometry information is modified to obtain modified three-dimensional facial geometry information. The spatial feature point information set corresponding to the aforementioned modified three-dimensional facial geometry information is rendered to obtain the de-occluded facial image information of the corresponding target object.

[0009] Secondly, some embodiments of this disclosure provide a real-time mixed reality head-mounted display occlusion removal device, including a generation unit configured to generate implicit encoded information based on a preset facial image information group using an encoder included in a pre-trained occlusion removal 3D facial information generation model; and an execution unit configured to perform the following steps using a decoder included in the pre-trained occlusion removal 3D facial information generation model: determining lower half-face deformation information corresponding to the lower half-face geometry information based on third preset feature constraint information, the aforementioned implicit encoded information, and lower half-face geometry information included in preset 3D facial geometry information; and determining lower half-face deformation information corresponding to the aforementioned lower half-face geometry information based on mapping association weight information, first preset feature constraint information, and the aforementioned preset 3D facial geometry information. The spatial feature point information set corresponding to the information is used to generate a spatial feature point information group corresponding to the aforementioned preset three-dimensional facial geometry information. Based on the aforementioned implicit encoding information, target weight, key geometric information of the upper half of the face included in the aforementioned spatial feature point information group, and upper half of the face included in the aforementioned preset three-dimensional facial geometry information, upper half of the face deformation information corresponding to the aforementioned upper half of the face geometry information is generated. Based on the aforementioned upper half of the face deformation information and the aforementioned lower half of the face deformation information, the aforementioned preset three-dimensional facial geometry information is modified to obtain modified three-dimensional facial geometry information. The spatial feature point information set corresponding to the aforementioned modified three-dimensional facial geometry information is rendered to obtain the de-occluded facial image information of the corresponding target object.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first or second aspect.

[0012] The above-described embodiments of this disclosure have the following beneficial effects: the real-time mixed reality head-mounted display occlusion removal method of some embodiments of this disclosure can improve the coherence and stability of obtaining facial dynamic changes, thereby improving the accuracy of occlusion removal, and can reduce the computing resources consumed by occlusion removal and shorten the time consumption, thereby improving the real-time performance of facial image dynamic changes and improving the rendering effect, thereby enhancing the user's interactive experience. The reasons for the poor coherence and stability of facial dynamic changes, low accuracy of occlusion removal, high computational resource consumption and long processing time for occlusion removal, poor real-time performance of facial image dynamic changes, poor rendering effects, and poor user interaction experience are as follows: When using an encoder-decoder architecture to extract visible region features to generate facial images, the lack of consistency between frames leads to poor coherence and stability of facial dynamic changes, resulting in low accuracy of occlusion removal. Furthermore, the high processing time for single-frame images leads to poor real-time performance of facial image dynamic changes, resulting in a poor user interaction experience. Additionally, occlusion removal using only two-dimensional features results in a limited field of view for capturing facial features from users wearing head-mounted displays, making it impossible to meet the requirements of binocular rendering and leading to poor rendering effects. Finally, using geometric reconstruction-based occlusion removal methods relies on 3D scanners, resulting in high computational complexity and high computational resource consumption for occlusion removal. Based on this, the real-time mixed reality head-mounted display de-occlusion method of some embodiments of this disclosure firstly generates implicit coding information based on a preset facial image information set by an encoder included in a pre-trained de-occlusion 3D facial information generation model. Thus, the implicit coding corresponding to the preset facial image information set can be obtained. Then, the decoder included in the pre-trained de-occlusion 3D facial information generation model performs the following steps: First, based on third preset feature constraint information, the implicit coding information, and the lower half-face geometry information included in the preset 3D facial geometry information, the lower half-face deformation information corresponding to the lower half-face geometry information is determined. Thus, the lower half-face deformation corresponding to the lower half-face geometry information can be obtained. Then, based on mapping association weight information, first preset feature constraint information, and the spatial feature point information set corresponding to the preset 3D facial geometry information, a spatial feature point information set corresponding to the preset 3D facial geometry information is generated. Thus, the spatial feature point information set corresponding to the preset 3D facial geometry information can be downsampled to obtain the spatial feature point information set. Secondly, based on the aforementioned implicit encoding information, target weights, the key geometric information of the upper face included in the aforementioned spatial feature point information group, and the upper face geometric information included in the aforementioned preset three-dimensional facial geometric information, upper face deformation information corresponding to the aforementioned upper face geometric information is generated. Thus, the upper face deformation variable corresponding to the aforementioned upper face geometric information can be obtained.Subsequently, based on the aforementioned upper and lower facial deformation information, the preset 3D facial geometry is modified to obtain modified 3D facial geometry. Thus, the preset 3D facial geometry can be modified to obtain the modified preset 3D facial geometry. Finally, the set of spatial feature points corresponding to the modified 3D facial geometry is rendered to obtain the de-occluded facial image information of the target object. This method does not use an encoder-decoder architecture to extract visible region features to generate facial images; instead, it uses the occluded image as direct input, without relying on a 3D scanner. Furthermore, it introduces a sampling method based on the correlation weights of the upper and lower face halves, allowing sampling of Gaussian points involved in the calculation based on the correlation weights of the 3D Gaussians of the upper and lower face halves. This reduces computational complexity, thereby reducing computational resources consumed in de-occlusion and shortening the processing time. Also, because it is based on facial image information groups, the preset 3D facial geometry is deformed to achieve de-occlusion of the facial image. The aforementioned preset 3D facial geometric information can represent the Gaussian model of the face under the initial default expression and pose in standard space. Therefore, its spatial continuity allows for multi-view rendering capabilities, thus improving rendering quality. Furthermore, mapping this preset 3D facial geometric information to a preset modeling deformation space considers the overall integrity of facial deformation and the influence of changes in the 3D Gaussian parameters when constructing the mapping relationship, thereby improving the overall integrity of facial deformation and enhancing the accuracy of generating de-occluded images. This improves the coherence and stability of capturing dynamic facial changes, further enhancing de-occlusion accuracy, reducing computational resources and time required for de-occlusion, improving the real-time performance of dynamic facial image changes, and enhancing rendering quality, ultimately improving the user's interactive experience. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the real-time mixed reality head-mounted display de-occlusion method according to the present disclosure;

[0015] Figure 2 This is a schematic diagram of the structure of some embodiments of the real-time mixed reality head-mounted display de-occlusion device according to the present disclosure;

[0016] Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0017] Figure 4 This is a schematic diagram of facial image information, preset three-dimensional facial geometry information, and modified three-dimensional facial geometry information in some embodiments of the real-time mixed reality head-mounted display de-occlusion method according to the present disclosure;

[0018] Figure 5 This is a schematic diagram comparing de-occluded facial image information with a standard image in some embodiments of the real-time mixed reality head-mounted display de-occlusion method according to this disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 A flow 100 of some embodiments of a real-time mixed reality head-mounted display occlusion removal method according to the present disclosure is shown. The real-time mixed reality head-mounted display occlusion removal method includes the following steps:

[0026] Step 101: Using the encoder included in the pre-trained de-occlusion 3D facial information generation model, implicit coded information is generated based on a preset facial image information group.

[0027] In some embodiments, the execution entity (e.g., a computing device) of the real-time mixed reality head-mounted display de-occlusion method can generate implicit encoded information based on a preset set of facial image information using an encoder included in a pre-trained de-occlusion 3D facial information generation model. The de-occlusion 3D facial information generation model may include a decoder and an encoder. The preset set of facial image information may represent the upper and lower half of the face image obtained after segmenting the facial image information of the target object. The preset set of facial image information may include lower half and upper half face image information. The lower half face image information may represent the lower half of the target object's face. The upper half face image information may represent the upper half of the target object's face. The target object may be a user wearing the head-mounted display device; the specific type of the head-mounted display device is not limited, for example, it may be AR glasses. The upper half of the face may represent the facial area of ​​the target object obscured by the head-mounted display device. For example, the upper half of the face may include the forehead, eyes, and bridge of the nose. The lower half of the face can represent the facial area of ​​the target object that is not obscured by the head-mounted display device. For example, the lower half of the face may include the tip of the nose, cheeks, and the facial area below the cheeks. The implicit encoded information can represent the facial expression features and pose of the lower half of the target object's face. The facial expression features can represent the facial expression of the target object. Here, the specific content of the facial expression features is not limited; for example, the facial expression feature can be a smile. The encoder can include four convolutional layers and one fully connected layer. Each of the four convolutional layers can include a 3×3 convolution (stride 2), instance normalization (InstanceNorm), a LeakyReLU activation function, and a channel attention module (SE module). The fully connected layer can include a linear transformation function and a normalization function. In practice, the execution entity can input the lower half of the face image information into the encoder to obtain the implicit encoded information corresponding to the lower half of the face image information.

[0028] Optionally, prior to step 101, the aforementioned execution entity may also perform the following steps:

[0029] The first step is to update the acquired facial image information to obtain modified facial image information. This modified facial image information can represent the face of the target object in an occluded state. The occlusion of the target object can represent the state where the user's face is obscured by the worn head-mounted display device. The modified facial image information can represent the image after background removal of the aforementioned facial image information. In practice, the execution entity can use the MODNet model to directly perform end-to-end background removal on the aforementioned facial image information through a target decomposition strategy to obtain the modified facial image information. Figure 4 The occluded image shown is a schematic diagram of the facial image information mentioned above.

[0030] The second step involves segmenting the altered facial image information based on the preset occlusion region information to obtain a set of facial image information as the preset facial image information set. The preset occlusion region information characterizes the facial area of ​​the target object that is obscured by the head-mounted display device. In practice, the executing entity can use the Cascaded Regression Tree (ERT) and OpenCV from the dlib library to segment the altered facial image information based on the preset occlusion region information to obtain the facial image information set.

[0031] Step 102, using the decoder included in the pre-trained de-occluded 3D facial information generation model, perform the following steps:

[0032] Step 1021: Based on the third preset feature constraint information, implicit coding information, and the lower half face geometric information included in the preset three-dimensional facial geometric information, determine the lower half face deformation information corresponding to the lower half face geometric information.

[0033] In some embodiments, the execution entity can determine lower face deformation information corresponding to the lower face geometry based on the third preset feature constraint information, the implicit encoding information, and the lower face geometry information included in the preset three-dimensional facial geometry information. The lower face deformation information can characterize the position offset, scaling factor, and rotation increment of each three-dimensional Gaussian element in the lower face geometry information. The third preset feature constraint information can characterize a nonlinear mapping function constructed based on an MLP network for deforming the lower face geometry information. The third preset feature constraint information can include a position MLP, a scaling MLP, and a rotation MLP. The position MLP can include one input layer, three hidden layers, and one output layer. The width of the first hidden layer in the position MLP can be 256. The width of the second hidden layer in the position MLP can be 256. The width of the third hidden layer in the position MLP can be 256. The scaling MLP can include one input layer, two hidden layers, and one output layer. The width of the first hidden layer in the scaling MLP can be 256. The width of the second hidden layer in the scaling MLP can be 256. The rotation MLP can include one input layer, two hidden layers, and one output layer. The width of the first hidden layer in the rotation MLP can be 256. The width of the second hidden layer in the rotation MLP can be 256. The width can be the number of neurons per layer. The preset 3D facial geometry information can include upper and lower facial geometry information. The preset 3D facial geometry information can represent a 3D Gaussian model of a face initialized in standard space with default expression and default pose. The preset 3D facial geometry information can include various 3D Gaussian units. The 3D Gaussian units in the above-mentioned 3D Gaussian units can represent the local geometry (position, scaling, rotation) and appearance attributes (color, transparency) of the preset 3D facial geometry information. The lower facial geometry information can represent the lower half Gaussian model of the corresponding lower half face initialized in standard space with default expression and default pose. The lower half Gaussian model can represent the various 3D Gaussian units corresponding to the above-mentioned lower facial geometry information. The decoder can include a 3D Gaussian deformation field and a rendering layer. The input to the aforementioned three-dimensional Gaussian deformation field can be implicitly encoded information, and the output of the aforementioned three-dimensional Gaussian deformation field can be modified three-dimensional facial geometry information. In practice, the aforementioned decoder can input the aforementioned lower half facial geometry information and the aforementioned implicitly encoded information into the aforementioned third preset feature constraint information to obtain the lower half facial deformation variables. As an example, the aforementioned third preset feature constraint information can be:

[0034] dG lower (dX lower ,dS lower ,dR lower ) = MLPs(X0,lower ,θ).

[0035] Among them, dG lower This can characterize the deformation variables of the aforementioned lower half Gaussian model. dX lower This can characterize the positional offset of the three-dimensional Gaussian elements in the aforementioned lower half facial geometry. dS lower This can be used to characterize the scaling factor of the three-dimensional Gaussian elements in the aforementioned lower half facial geometry. dR lower The rotation increment of the three-dimensional Gaussian elements in the aforementioned lower half-face geometry information can be represented. MLPs can represent the aforementioned third preset feature constraint information. X 0，lower The positional parameters can represent the geometric information of the lower half of the face in standard space. θ can represent the implicitly encoded information mentioned above.

[0036] Optionally, before step 1021, the execution entity may also generate preset three-dimensional facial geometric information based on the preset model information and the three-dimensional Gaussian deformation field included in the decoder. The preset model information may be a 3DMM model. The upper facial geometric information may represent the upper face Gaussian model corresponding to the default expression and pose initialized in standard space. The upper face Gaussian model may represent each three-dimensional Gaussian element corresponding to the upper facial geometric information. In practice, the decoder may initialize the preset three-dimensional facial geometric information using the 3DMM model to obtain the initialized preset three-dimensional facial geometric information.

[0037] Step 1022: Generate a spatial feature point information group corresponding to the preset three-dimensional facial geometry information based on the mapping association weight information, the first preset feature constraint information, and the spatial feature point information set corresponding to the preset three-dimensional facial geometry information.

[0038] In some embodiments, the execution entity can generate a spatial feature point information group corresponding to the preset three-dimensional facial geometry based on the mapping association weight information, the first preset feature constraint information, and the spatial feature point information set corresponding to the preset three-dimensional facial geometry. The first preset feature constraint information can represent a nonlinear mapping function constructed based on an MLP network for performing position offset, scaling, and rotation processing on the preset three-dimensional facial geometry to transform it into a 10-dimensional feature vector. The first preset feature constraint information can include one input layer, two hidden layers, and one output layer. The width of the first hidden layer included in the first preset feature constraint information can be 128. The width of the second hidden layer included in the first preset feature constraint information can be 256. The mapping association weight information can represent the Euclidean norm between the spatial feature point information set of the upper half of the facial geometry and the spatial feature point information set of the lower half of the facial geometry. The spatial feature point information set corresponding to the preset three-dimensional facial geometry can represent all three-dimensional Gaussian elements within the preset three-dimensional facial geometry. The spatial feature point information group can include key geometric information of the upper half of the face and key geometric information of the lower half of the face. The aforementioned key geometric information of the upper half of the face can represent the individual three-dimensional Gaussian elements obtained after downsampling the aforementioned upper half of the face's geometric information. Similarly, the aforementioned key geometric information of the lower half of the face can represent the individual three-dimensional Gaussian elements obtained after downsampling the aforementioned lower half of the face's geometric information. In practice, the aforementioned execution entity can also perform the following steps through the three-dimensional Gaussian deformation field included in the decoder:

[0039] In the process of adopting technical solutions to address the aforementioned technical problems, the following technical problem often arises: Directly removing occlusion from the complete 3D Gaussian units of a face obtained by a 3D scanner results in a large number of Gaussian units, leading to high computational resource consumption and long processing time, and a poor user experience. A conventional solution to this second technical problem is to randomly select 3D Gaussian units of the face for occlusion removal. However, considering the drawbacks of randomly selecting 3D Gaussian units for occlusion removal, and leveraging the advantages of our institution in processing complete 3D Gaussian units of the face, we have decided to adopt the following solution:

[0040] In some optional implementations of certain embodiments, the aforementioned three-dimensional Gaussian deformation field can generate a spatial feature point information group corresponding to the aforementioned preset three-dimensional facial geometry information through the following steps: based on the mapping association weight information, the first preset feature constraint information, and the spatial feature point information set corresponding to the aforementioned preset three-dimensional facial geometry information.

[0041] The first step is to determine the lower half-face feature vector and the upper half-face feature vector corresponding to the lower half-face geometry based on the first preset feature constraint information. The lower half-face feature vector can represent the 10-dimensional feature vector of the lower half-face geometry. The upper half-face feature vector can represent the 10-dimensional feature vector of the upper half-face geometry. In practice, the three-dimensional Gaussian deformation field can first input the position (three-dimensional coordinates), scaling factor, and rotation quaternion corresponding to the preset three-dimensional facial geometry into the first preset feature constraint information to obtain the 10-dimensional feature vectors corresponding to the lower half-face geometry and the upper half-face geometry. The position (three-dimensional coordinates) of the preset three-dimensional facial geometry can represent the position (three-dimensional coordinates) of the lower half-face geometry and the upper half-face geometry. As an example, the first preset feature constraint information can be:

[0042]

[0043] in, A 10-dimensional feature vector can represent the geometric information of the lower half of the face, and a 10-dimensional feature vector can represent the geometric information of the upper half of the face. MLP gm The first preset feature constraint information can be represented. X0 can represent the position (3D coordinates) of the preset 3D facial geometry information. S0 can represent the scaling factor corresponding to the preset 3D facial geometry information. R0 can represent the rotation quaternion corresponding to the preset 3D facial geometry information.

[0044] The second step involves the aforementioned three-dimensional Gaussian deformation field determining mapping and association weight information for the corresponding preset three-dimensional facial geometry based on the lower half-face feature vector corresponding to the lower half-face geometry and the upper half-face feature vector corresponding to the upper half-face geometry. In practice, the aforementioned three-dimensional Gaussian deformation field can input the lower half-face feature vector and the upper half-face feature vector into a first preset function to obtain the mapping and association weight information. As an example, the first preset function can be:

[0045]

[0046] Among them, W corr Weight information can be associated with the mapping. X·x 2 It can be the square of the x-component of the position parameter X in the aforementioned upper half-face feature vector and the aforementioned lower half-face feature vector. X·y 2 It can be the square of the y-component of the position parameter X in the aforementioned upper half-face feature vector and the aforementioned lower half-face feature vector. X·z 2It can be the square of the z-component of the position parameter X in the above upper half face feature vector and the above lower half face feature vector.

[0047] The third step involves downsampling the spatial feature point information set corresponding to the aforementioned preset 3D facial geometry information and the aforementioned mapping association weight information, to obtain a spatial feature point information group. The preset selection condition can be that the index of the spatial feature point information in the aforementioned preset 3D facial geometry information set is greater than a preset index. The specific content of the preset index is not limited here; for example, the preset index can be 512. In practice, firstly, the aforementioned 3D Gaussian deformation field can sort the spatial feature point information in the aforementioned spatial feature point information set in descending order according to the aforementioned mapping association weight information, to obtain a sorted spatial feature point information set. Then, the spatial feature point information in the sorted spatial feature point information set that satisfies the aforementioned preset selection condition is determined as the spatial feature point information group.

[0048] The above-described technical solution, as an inventive point of this disclosure, solves technical problem two: "The processing of 3D Gaussian elements of a complete face consumes a large amount of computational resources and takes a long time, resulting in a poor user experience." The reasons for this are as follows: Directly removing occlusion from 3D Gaussian elements of a complete face obtained through a 3D scanner results in a large number of 3D Gaussian elements, leading to high computational resource consumption and long processing time, thus negatively impacting the user experience. Solving these factors reduces the computational resources required for processing 3D Gaussian elements of a complete face, shortens the processing time, and improves the user experience. To achieve this effect, the real-time mixed reality head-mounted display occlusion removal method of this disclosure first determines the mapping association weight information of the preset 3D facial geometry information based on the lower half-face feature vector and the upper half-face feature vector. Therefore, the spatial positional dependency between the spatial feature point information set of the upper half of the face and the spatial feature point information set of the lower half of the face can be obtained. Then, the spatial feature point information set corresponding to the preset 3D facial geometry is filtered according to the mapping association weight information, thereby obtaining the key 3D Gaussian units that contribute the most to the mapping of the upper and lower face. Subsequently, occlusion removal processing is performed based on the filtered spatial feature point information set. This reduces the number of 3D Gaussian units involved in the occlusion removal process, and since the occlusion removal processing is based on the key 3D Gaussian units that contribute the most to the mapping of the upper and lower face, the impact on occlusion removal quality is reduced. Furthermore, it reduces the computational resources required for occlusion removal and shortens the processing time, thereby improving the user's interactive experience.

[0049] Optionally, prior to step 1023, the aforementioned execution entity may also perform the following steps using the three-dimensional Gaussian deformation field included in the decoder:

[0050] The first step is to determine the facial feature weight matrix by multiplying the transpose of the lower half facial feature vector with the upper half facial feature vector.

[0051] The second step is to normalize the facial feature weight matrix to obtain the facial feature weight vector. This facial feature weight vector represents the normalized dependency weights of the upper half of the facial feature vector on the lower half of the facial feature vector. In practice, the three-dimensional Gaussian deformation field can be normalized using a row-wise Softmax function to the facial feature weight matrix.

[0052] The third step involves determining the upper and lower face weight vectors corresponding to the aforementioned preset 3D facial geometry information based on the second preset feature constraint information. The upper face weight vector represents the weight vectors representing the influence of each parameter of the upper face Gaussian model on deformation propagation. The lower face weight vector represents the weight vectors representing the influence of each parameter of the lower face Gaussian model on deformation propagation. These parameters may include position (3D coordinates), scaling factor, and rotation quaternion. The second preset feature constraint information represents the nonlinear mapping function constructed based on the MLP network for generating the parameter components corresponding to the aforementioned preset 3D facial geometry information. These parameter components may include position offset, scaling factor, and rotation increment. The second preset feature constraint information may include one input layer, two hidden layers, and one output layer. The width of the first hidden layer included in the second preset feature constraint information can be 128. The width of the second hidden layer included in the second preset feature constraint information can be 256. In practice, the aforementioned three-dimensional Gaussian deformation field can input the aforementioned preset three-dimensional facial geometry information into the second preset feature constraint information to obtain the upper half-face weight vector and lower half-face weight vector corresponding to the aforementioned preset three-dimensional facial geometry information. As an example, the aforementioned second preset feature constraint information can be:

[0053] W pm =MLP pm (G0).

[0054] Among them, W pm It can represent the upper half-face weight vector and the lower half-face weight vector. MLP pm It can represent the second preset feature constraint information mentioned above. G0 can represent the preset three-dimensional facial geometry information mentioned above.

[0055] The fourth step is to determine the target weight by multiplying the above facial feature weight vector, the above upper face weight vector, and the above lower face weight vector.

[0056] Step 1023: Generate upper face deformation information corresponding to the upper face geometric information based on implicit coding information, target weight, key geometric information of the upper face included in the spatial feature point information group, and upper face geometric information included in the preset three-dimensional facial geometric information.

[0057] In some embodiments, the execution entity can generate upper-half facial deformation information corresponding to the aforementioned upper-half facial geometry based on the implicit encoded information, target weights, the upper-half facial key geometric information included in the aforementioned spatial feature point information group, and the upper-half facial geometry information included in the aforementioned preset three-dimensional facial geometry information. The upper-half facial deformation information can characterize the position offset, scaling factor, and rotation increment of each three-dimensional Gaussian element in the upper-half facial geometry information. In practice, the execution entity can perform the following steps using the three-dimensional Gaussian deformation field included in the decoder:

[0058] In some optional implementations of certain embodiments, the aforementioned three-dimensional Gaussian deformation field can generate upper-half facial deformation information corresponding to the aforementioned upper-half facial geometry information through the following steps: based on the aforementioned implicit coding information, target weights, the upper-half facial key geometric information included in the aforementioned spatial feature point information group, and the upper-half facial geometric information included in the aforementioned preset three-dimensional facial geometric information.

[0059] The first step involves generating distance weights between the spatial feature points of the upper face and the key geometric information of the upper face, based on the aforementioned key geometric information and the key geometric information of the upper face. The spatial feature points can represent three-dimensional Gaussian elements. The distance weights represent the spatial proximity between the three-dimensional Gaussian model of the upper face and the key geometric information of the upper face. In practice, firstly, the three-dimensional Gaussian deformation field can determine the Euclidean distance between the upper face and the key geometric information of the upper face. Then, the Euclidean distance and a minimum positive number are input into a second preset function to obtain the distance weights. As an example, the second preset function can be:

[0060]

[0061] Among them, G key，j It can be the j-th Gaussian element in the key geometric information of the upper half of the face. G upper，i It can be the i-th Gaussian element in the geometric information of the upper half of the face mentioned above. W dist，ii This can be used as the distance weight between the i-th Gaussian element in the key geometric information of the upper half of the face and the j-th Gaussian element in the aforementioned geometric information of the upper half of the face. Distance(G) key,j G upper,i ) can be the Euclidean distance between the i-th Gaussian element in the key geometric information of the upper half of the face and the j-th Gaussian element in the aforementioned geometric information of the upper half of the face. eps can be a very small positive number.

[0062] The second step is to normalize the aforementioned distance weights to obtain the target distance weights. These target distance weights characterize the normalized distance weights. In practice, the aforementioned three-dimensional Gaussian deformation field can be normalized using row direction normalization to obtain the target distance weights.

[0063] The third step involves generating upper-half facial deformation information corresponding to the aforementioned upper-half facial geometry, based on the target distance weight, target weight, implicit encoding information, fourth preset feature constraint information, upper-half facial geometry, and upper-half facial key geometry. The fourth preset feature constraint information can characterize a nonlinear mapping function constructed based on an MLP network for deforming the upper-half facial geometry. This fourth preset feature constraint information may include one input layer, three hidden layers, and one output layer. The width of the first hidden layer, the width of the second hidden layer, and the width of the third hidden layer can all be 256. In practice, the three-dimensional Gaussian deformation field can input the target distance weight, target weight, implicit encoding information, upper-half facial geometry, and upper-half facial key geometry into the fourth preset feature constraint information to obtain upper-half facial deformation information. As an example, the fourth preset feature constraint information can be:

[0064] dG upper =MLP map (G 0，upper ,θ,W dist ×W pm，upper ×W gm ×W pm，lower ×(dG key G key )).

[0065] Among them, dG upper It can characterize the deformation of the upper half of the 3D Gaussian model. MLP map This can characterize the aforementioned fourth preset feature constraint information. G 0，upper It can represent a 3D Gaussian model of the upper half of the face. W pm，upper A weight vector that can represent the geometric information of the upper half of the face. W gm This can represent the facial feature weight vector mentioned above. W pm，lower This can be represented by a weight vector dG that encapsulates the geometric information of the lower half of the face. key Deformation variables that can characterize key geometric information of the upper face. G key It can represent key geometric information of the upper face.

[0066] Step 1024: Based on the deformation information of the upper half of the face and the deformation information of the lower half of the face, the preset three-dimensional facial geometry information is modified to obtain the modified three-dimensional facial geometry information.

[0067] In some embodiments, the aforementioned three-dimensional Gaussian deformation field can modify the preset three-dimensional facial geometry information based on the deformation information of the upper half of the face and the deformation information of the lower half of the face, thereby obtaining modified three-dimensional facial geometry information. This allows the preset three-dimensional facial geometry information to be mapped to a preset modeling deformation space. The preset modeling deformation space can represent the deformation space related to facial dynamic features in the three-dimensional Gaussian deformation field. The modified three-dimensional facial geometry information can represent the Gaussian model mapped to the deformation space. The facial dynamic features can represent facial expression features and pose. Figure 4 The standard spatial Gaussian model shown is a schematic diagram of the aforementioned preset three-dimensional facial geometry. Figure 4 The Gaussian model of the deformation space shown is a schematic diagram of the above-mentioned alteration of three-dimensional facial geometry.

[0068] In some optional implementations of certain embodiments, the aforementioned three-dimensional Gaussian deformation field can be modified by altering the preset three-dimensional facial geometry information based on the deformation information of the upper half of the face and the deformation information of the lower half of the face through the following steps:

[0069] The first step involves generating pose deformation variable information corresponding to the aforementioned preset 3D facial geometry information based on the fifth preset feature constraint information and the aforementioned implicit encoding information. The pose deformation variable information can characterize the position offset, rotation increment, and scaling factor of the preset 3D facial geometry information. The fifth preset feature constraint information can characterize a nonlinear mapping function constructed based on an MLP network for deforming the aforementioned preset 3D facial geometry information. The fifth preset feature constraint information can include one input layer, two hidden layers, and one output layer. The width of the first hidden layer included in the fifth preset feature constraint information can be 128. The width of the second hidden layer included in the fifth preset feature constraint information can be 256. In practice, the aforementioned 3D Gaussian deformation field can input the aforementioned implicit encoding information into the fifth preset feature constraint information to obtain the pose deformation variable information. As an example, the aforementioned fifth preset feature constraint information can be:

[0070] dG global (dX global ,dS global ,dR global ) = MLP global (θ).

[0071] Among them, dG global This can characterize the above pose variables. dXglobal This can characterize the positional offset of the aforementioned preset three-dimensional facial geometry information. dS global It can represent the scaling factor of a preset 3D facial geometry. dR global It can represent the rotational increment of predefined 3D facial geometry information. MLP global It can characterize the fifth preset feature constraint information mentioned above.

[0072] The second step involves modifying the preset three-dimensional facial geometry based on the aforementioned pose deformation information, upper facial deformation information, and lower facial deformation information, to obtain modified three-dimensional facial geometry. In practice, the aforementioned three-dimensional Gaussian deformation field can input the aforementioned pose deformation information, upper facial deformation information, and lower facial deformation information into a third preset function to obtain the modified three-dimensional facial geometry. As an example, the third preset function can be:

[0073]

[0074] Where X represents the positional parameters of the pre-deformed 3D facial geometry. X0 represents the positional parameters of the pre-deformed 3D facial geometry. dX represents the positional offset of the pre-deformed 3D facial geometry. S represents the scaling parameters of the pre-deformed 3D facial geometry. S0 represents the scaling parameters of the pre-deformed 3D facial geometry. dS represents the scaling factor of the pre-deformed 3D facial geometry. R represents the rotation quaternion of the pre-deformed 3D facial geometry. R0 can be the rotation quaternion before deformation. dR represents the rotation increment of the pre-deformed 3D facial geometry. It can be a quaternion multiplication operator.

[0075] Step 1025: Render the set of spatial feature points corresponding to the changed 3D facial geometry information to obtain the occluded facial image information of the corresponding target object.

[0076] In some embodiments, the decoder can render the set of spatial feature points corresponding to the altered 3D facial geometry to obtain de-occluded facial image information of the target object. The set of spatial feature points corresponding to the altered 3D facial geometry can represent all 3D Gaussian units within the altered 3D facial geometry. The de-occluded facial image information can represent the facial image after de-occlusion processing of the facial image information. In practice, the rendering layer included in the decoder can perform rasterization rendering processing on the altered 3D facial geometry using a 3D Gaussian sputtering pipeline to obtain a rendered de-occluded RGB image. The rendering layer can take the set of spatial feature points corresponding to the altered 3D facial geometry as input and the de-occluded facial image information as output. Figure 5 As shown, Figure 5 This can be used to represent a comparison diagram of the de-occlusion effect (VR left eye) image, the de-occlusion effect (VR right eye) image, and a standard image. Figure 5 The occlusion effect (VR left eye) and de-occlusion effect (VR right eye) diagrams shown can both represent the above-mentioned de-occluded facial image information.

[0077] In the process of adopting technical solutions to solve the above-mentioned technical problems, the following technical problem three often arises: During the training of the occlusion-removing 3D facial information generation model, due to the regional division of the 3D Gaussian units included in the upper and lower facial geometric information, some 3D Gaussian units may cross the boundary into the opposite region, resulting in low stability of the occlusion-removing 3D facial information generation model training, poor occlusion quality of the model, and consequently, a poor user experience. The conventional solution to this technical problem three is to set a fixed boundary threshold to forcibly limit the splitting range of the 3D Gaussian units. However, considering the drawbacks of setting a fixed boundary threshold to forcibly limit the splitting range of the 3D Gaussian units, and also taking into account the spatial constraints available to the inventors' institution, we decided to adopt the following solution:

[0078] Optionally, the aforementioned occlusion-removing 3D facial information generation model can be trained by the aforementioned execution entity through the following steps:

[0079] The first step is to obtain a sample set. This sample set can include video frame information from publicly available datasets and target user video frame information from a self-built dataset. The video frame information can be individual images obtained by splitting videos from the Nersemble dataset. The target user video frame information can be individual images obtained by directly splitting the acquired user videos. Each image in the sample set is an unoccluded face image.

[0080] The second step involves randomly selecting video frame information from the publicly available dataset and target user video frame information from the self-built dataset to obtain training set image frame information and test set image frame information. The training set image frame information represents 80% of the images randomly selected from the publicly available dataset and the self-built dataset. The test set image frame information represents 20% of the images randomly selected from the publicly available dataset and the self-built dataset.

[0081] The third step involves selecting samples from the image frame information in the training set and performing the following training steps:

[0082] The first sub-step involves inputting the image frame information of at least one sample from the training set image frame information into the encoder of the initial neural network to obtain the implicit encoding information of the lower half of the face image information corresponding to the sample.

[0083] The second sub-step involves inputting the implicit encoded information corresponding to the aforementioned samples into the decoder of the initial neural network to obtain the de-occluded facial image information corresponding to the samples. The structure of the initial neural network is the same as that of the de-occluded 3D facial information generation model, and will not be repeated here. The initial neural network can take the implicit encoded information as input and the de-occluded facial image information as output.

[0084] The third sub-step generates loss data based on preset boundary threshold information, preset margin values, the ordinate coordinates of the upper and lower half of the face geometry in the modified 3D facial geometry, preset separation constraint information, preset color loss term, preset perception loss term, first preset weight coefficient, second preset weight coefficient, and third preset weight coefficient. The preset boundary threshold information represents the ordinate values ​​used to divide the upper and lower face regions. The upper and lower face regions can include the face region corresponding to the upper face geometry and the face region corresponding to the lower face geometry. The preset margin values ​​represent the maximum upward and downward movement of the preset boundary threshold information. The specific content of the preset boundary threshold information and the preset margin values ​​is not limited here; for example, the preset boundary threshold information can be y=5, and the preset margin value can be 2. The ordinate coordinate information represents the Y-axis coordinates corresponding to the key upper and lower face geometry. The loss data represents the loss value. The aforementioned preset separation constraint information may include an upper face separation constraint loss term and a lower face separation constraint loss term. The upper face separation constraint loss term can characterize the function that the maximum downward movement of the 3D Gaussian element corresponding to the upper face geometry controlled by the executing agent during model training is less than or equal to the aforementioned preset margin value. The lower face separation constraint loss term can characterize the function that the maximum upward movement of the 3D Gaussian element corresponding to the lower face geometry controlled by the executing agent during model training is less than or equal to the aforementioned preset margin value. The aforementioned preset color loss term can characterize the basic reconstruction loss in the standard Gaussian rendering process. The basic reconstruction loss can characterize the deviation between the rendered de-occluded RGB image and the complete face supervision image in terms of basic visual features such as color and brightness. Here, the specific content of the aforementioned basic visual features is not limited; for example, basic visual features can be the color depth of facial skin. The aforementioned complete face supervision image can characterize the unoccluded face images included in the aforementioned sample set. The aforementioned preset color loss term includes an L1 loss (MAE) function and a structural similarity loss function. The aforementioned preset perceptual loss term can characterize the perceptual loss function. The aforementioned first preset weight coefficient can be the weight coefficient corresponding to the aforementioned preset color loss term. Here, the specific content of the aforementioned first preset weight coefficient is not limited; for example, the first preset weight coefficient can be 0.4. The aforementioned second preset weight coefficient can be the weight coefficient corresponding to the aforementioned preset perceptual loss term. Here, the specific content of the aforementioned second preset weight coefficient is not limited; for example, the second preset weight coefficient can be 0.1.The aforementioned third preset weight coefficient can be the weight coefficient of the sum of the aforementioned upper half face separation constraint loss term and the aforementioned lower half face separation constraint loss term. Here, the specific content of the aforementioned third preset weight coefficient is not limited; for example, the third preset weight coefficient can be 0.01. In practice, firstly, the aforementioned executing entity can input the aforementioned preset boundary threshold information, the aforementioned preset margin value, the aforementioned vertical axis coordinate information of the upper half face geometry information that changes the three-dimensional face geometry information, and the aforementioned vertical axis coordinate information of the lower half face geometry information that changes the three-dimensional face geometry information into the preset separation constraint information to obtain the upper half face separation constraint loss term and the lower half face separation constraint loss term. As an example, the aforementioned preset separation constraint information (the aforementioned preset separation constraint information includes the upper half face separation constraint loss term which can be (1), and the aforementioned preset separation constraint information includes the lower half face separation constraint loss term which can be (2)) can be:

[0085] (1)L upper =ReLU(-(G upper ·y-(y t -m)))

[0086] (2)L lower =ReLU(-(G lower ·y-(y t +m))).

[0087] Among them, L upper This can characterize the loss term for the separation constraint of the upper half of the face mentioned above. L lower This can characterize the lower half-face separation constraint loss term mentioned above. G upper .y can represent the ordinate of each three-dimensional Gaussian element corresponding to the geometric information of the upper half of the face mentioned above. G lower .y can represent the ordinate of each three-dimensional Gaussian element corresponding to the geometric information of the lower half of the face mentioned above. t This can represent the aforementioned preset boundary threshold information. m can represent the aforementioned preset boundary value.

[0088] Then, the aforementioned upper face separation constraint loss term, lower face separation constraint loss term, preset color loss term, preset perception loss term, first preset weight coefficient, second preset weight coefficient, and third preset weight coefficient are input into the fourth preset function to obtain the loss data. As an example, the fourth preset function can be:

[0089] L=λ rgb L rgb +λ vgg L vgg +λ split (L upper +Ll ower ).

[0090] Where L can represent the total loss value between the input image frame information and the output de-occluded RGB image. λ rgb This can represent the weighting coefficients of the aforementioned preset color loss term. L rgb This can characterize the aforementioned preset color loss term. λ vgg The weighting coefficients can characterize the aforementioned pre-defined perceptual loss term. L vgg This can characterize the aforementioned pre-defined perceptual loss term. λ split The weighting coefficients can represent the sum of the above upper half face separation constraint loss term and the above lower half face separation constraint loss term.

[0091] The fourth sub-step involves adjusting the initial neural network to obtain the adjusted initial neural network. In practice, the aforementioned execution entity can adjust the network parameters of the initial neural network using the backpropagation algorithm (BP algorithm) and gradient descent method based on the aforementioned loss data. Here, the specific content of the gradient descent method is not limited; for example, it can be the mini-batch gradient descent algorithm.

[0092] The fifth sub-step involves determining the initial neural network as a trained de-occlusion model in response to the determination that the loss data satisfies preset constraints. This de-occlusion model may include an encoder and a decoder. The preset constraints can be defined as the number of epochs in which the network parameters of the initial neural network are adjusted based on the loss data. This preset number of epochs can be a manually set value. The specific value of the preset number of epochs is not limited here; for example, it could be 10000.

[0093] Optionally, the execution entity may also adjust the network parameters of the initial neural network and use unused samples to form a sample set in response to determining that the loss data does not meet the preset constraints, and then use the adjusted initial neural network to perform the above training steps again.

[0094] The above-described technical solution, as an inventive point of this disclosure, solves technical problem three: "the training stability of the occluded 3D facial information generation model is low, the occlusion quality of the model is poor, and consequently, the user's interactive experience is poor." The reasons for the low training stability, poor occlusion quality, and poor user interactive experience of the occluded 3D facial information generation model are as follows: During the training process of the occluded 3D facial information generation model, due to the regional division of the 3D Gaussian units included in the upper and lower facial geometric information, some 3D Gaussian units may cross the boundary into the opposite region, resulting in low training stability, poor occlusion quality, and consequently, a poor user interactive experience. Solving these factors can improve the training stability of the occluded 3D facial information generation model, improve the occlusion quality of the model, and enhance the user's interactive experience. To achieve this effect, the real-time mixed reality head-mounted display occlusion removal method disclosed herein firstly inputs the aforementioned preset boundary threshold information, the aforementioned preset margin values, the ordinate coordinates of the upper half of the face geometry information and the lower half of the face geometry information (which have been modified in the 3D facial geometry information) into preset separation constraint information. This yields upper half-face separation constraint loss terms and lower half-face separation constraint loss terms. The aforementioned preset boundary threshold information represents the ordinate values ​​used to divide the upper and lower half of the face regions. The aforementioned preset margin values ​​represent the maximum upward and downward movement of the aforementioned preset boundary threshold information. Then, the aforementioned upper half-face separation constraint loss terms, the aforementioned lower half-face separation constraint loss terms, a preset color loss term, a preset perception loss term, a first preset weight coefficient, a second preset weight coefficient, and a third preset weight coefficient are input into a fourth preset function. This yields loss data. Afterwards, the initial neural network is adjusted. This yields the adjusted initial neural network. Finally, in response to the determination that the loss data meets the preset constraints, the initial neural network is identified as the trained de-occlusion model. This improves the stability of the de-occlusion 3D facial information generation model training and enhances the quality of model de-occlusion, thereby improving the user's interactive experience.

[0095] The above-described embodiments of this disclosure have the following beneficial effects: the real-time mixed reality head-mounted display occlusion removal method of some embodiments of this disclosure can improve the coherence and stability of obtaining facial dynamic changes, thereby improving the accuracy of occlusion removal, and can reduce the computing resources consumed by occlusion removal and shorten the time consumption, thereby improving the real-time performance of facial image dynamic changes and improving the rendering effect, thereby enhancing the user's interactive experience. The reasons for the poor coherence and stability of facial dynamic changes, low accuracy of occlusion removal, high computational resource consumption and long processing time for occlusion removal, poor real-time performance of facial image dynamic changes, poor rendering effects, and poor user interaction experience are as follows: When using an encoder-decoder architecture to extract visible region features to generate facial images, the lack of consistency between frames leads to poor coherence and stability of facial dynamic changes, resulting in low accuracy of occlusion removal. Furthermore, the high processing time for single-frame images leads to poor real-time performance of facial image dynamic changes, resulting in a poor user interaction experience. Additionally, occlusion removal using only two-dimensional features results in a limited field of view for capturing facial features from users wearing head-mounted displays, making it impossible to meet the requirements of binocular rendering and leading to poor rendering effects. Finally, using geometric reconstruction-based occlusion removal methods relies on 3D scanners, resulting in high computational complexity and high computational resource consumption for occlusion removal. Based on this, the real-time mixed reality head-mounted display de-occlusion method of some embodiments of this disclosure firstly generates implicit coding information based on a preset facial image information set by an encoder included in a pre-trained de-occlusion 3D facial information generation model. Thus, the implicit coding corresponding to the preset facial image information set can be obtained. Then, the decoder included in the pre-trained de-occlusion 3D facial information generation model performs the following steps: First, based on third preset feature constraint information, the implicit coding information, and the lower half-face geometry information included in the preset 3D facial geometry information, the lower half-face deformation information corresponding to the lower half-face geometry information is determined. Thus, the lower half-face deformation corresponding to the lower half-face geometry information can be obtained. Then, based on mapping association weight information, first preset feature constraint information, and the spatial feature point information set corresponding to the preset 3D facial geometry information, a spatial feature point information set corresponding to the preset 3D facial geometry information is generated. Thus, the spatial feature point information set corresponding to the preset 3D facial geometry information can be downsampled to obtain the spatial feature point information set. Secondly, based on the aforementioned implicit encoding information, target weights, the key geometric information of the upper face included in the aforementioned spatial feature point information group, and the upper face geometric information included in the aforementioned preset three-dimensional facial geometric information, upper face deformation information corresponding to the aforementioned upper face geometric information is generated. Thus, the upper face deformation variable corresponding to the aforementioned upper face geometric information can be obtained.Subsequently, based on the aforementioned upper and lower facial deformation information, the preset 3D facial geometry is modified to obtain modified 3D facial geometry. Thus, the preset 3D facial geometry can be modified to obtain the modified preset 3D facial geometry. Finally, the set of spatial feature points corresponding to the modified 3D facial geometry is rendered to obtain the de-occluded facial image information of the target object. This method does not use an encoder-decoder architecture to extract visible region features to generate facial images; instead, it uses the occluded image as direct input, without relying on a 3D scanner. Furthermore, it introduces a sampling method based on the correlation weights of the upper and lower face halves, allowing sampling of Gaussian points involved in the calculation based on the correlation weights of the 3D Gaussians of the upper and lower face halves. This reduces computational complexity, thereby reducing computational resources consumed in de-occlusion and shortening the processing time. Also, because it is based on facial image information groups, the preset 3D facial geometry is deformed to achieve de-occlusion of the facial image. The aforementioned preset 3D facial geometric information can represent the Gaussian model of the face under the initial default expression and pose in standard space. Therefore, its spatial continuity allows for multi-view rendering capabilities, thus improving rendering quality. Furthermore, mapping this preset 3D facial geometric information to a preset modeling deformation space considers the overall integrity of facial deformation and the influence of changes in the 3D Gaussian parameters when constructing the mapping relationship, thereby improving the overall integrity of facial deformation and enhancing the accuracy of generating de-occluded images. This improves the coherence and stability of capturing dynamic facial changes, further enhancing de-occlusion accuracy, reducing computational resources and time required for de-occlusion, improving the real-time performance of dynamic facial image changes, and enhancing rendering quality, ultimately improving the user's interactive experience.

[0096] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a real-time mixed reality head-mounted display occlusion removal method. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 2As shown, a real-time mixed reality head-mounted display de-occlusion device 200 according to some embodiments includes a generation unit 201 and an execution unit 202. The generation unit 201 is configured to generate implicit encoded information based on a preset facial image information set using an encoder included in a pre-trained de-occlusion 3D facial information generation model. The execution unit 202 is configured to perform the following steps using a decoder included in the pre-trained de-occlusion 3D facial information generation model: determining lower half-face deformation information corresponding to the lower half-face geometry information based on third preset feature constraint information, the implicit encoded information, and lower half-face geometry information included in preset 3D facial geometry information; and determining lower half-face deformation information based on mapping association weight information, first preset feature constraint information, and a set of spatial feature point information corresponding to the preset 3D facial geometry information. A spatial feature point information group corresponding to the aforementioned preset three-dimensional facial geometric information is generated; based on the aforementioned implicit encoding information, target weight, key geometric information of the upper half of the face included in the aforementioned spatial feature point information group, and upper half of the face included in the aforementioned preset three-dimensional facial geometric information, upper half facial deformation information corresponding to the aforementioned upper half facial geometric information is generated; based on the aforementioned upper half facial deformation information and the aforementioned lower half facial deformation information, the aforementioned preset three-dimensional facial geometric information is modified to obtain modified three-dimensional facial geometric information; the spatial feature point information set corresponding to the aforementioned modified three-dimensional facial geometric information is rendered to obtain the de-occluded facial image information of the corresponding target object.

[0098] It is understandable that the units and references described in the real-time mixed reality head-mounted display de-occlusion device 200 Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the occlusion removal device 200 and the units contained therein applied to a head-mounted display device, and will not be repeated here.

[0099] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0100] like Figure 3As shown, the electronic device 300 may include a processing unit 301 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0101] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0102] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0103] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0104] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0105] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: generate implicit encoded information based on a preset facial image information group using the encoder included in the pre-trained de-occlusion 3D facial information generation model; and perform the following steps using the decoder included in the pre-trained de-occlusion 3D facial information generation model: determine lower half-face deformation information corresponding to the lower half-face geometry information based on the third preset feature constraint information, the aforementioned implicit encoded information, and the lower half-face geometry information included in the preset 3D facial geometry information; and perform the following steps based on the mapping association weight information, the first preset feature constraint information, and the aforementioned preset 3D facial geometry information. The corresponding set of spatial feature points is used to generate a set of spatial feature points corresponding to the aforementioned preset three-dimensional facial geometry. Based on the aforementioned implicit encoding information, target weights, the key geometric information of the upper half of the face included in the aforementioned spatial feature point set, and the upper half of the face included in the aforementioned preset three-dimensional facial geometry, upper half facial deformation information corresponding to the aforementioned upper half facial geometry is generated. Based on the aforementioned upper half facial deformation information and the aforementioned lower half facial deformation information, the aforementioned preset three-dimensional facial geometry is modified to obtain modified three-dimensional facial geometry. The set of spatial feature points corresponding to the aforementioned modified three-dimensional facial geometry is rendered to obtain the de-occluded facial image information of the corresponding target object.

[0106] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor, for example, and can be described as a generation unit and an execution unit. The names of these units do not necessarily limit the unit itself; for example, a generation unit can also be described as "a unit that generates implicitly encoded information based on a preset set of facial image information, using an encoder included in a pre-trained de-occlusion 3D facial information generation model."

[0109] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0110] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for de-occlusion of a real-time mixed reality head-mounted display, comprising: The encoder in the pre-trained de-occlusion 3D facial information generation model generates implicit coded information based on a preset set of facial image information. The following steps are performed using the decoder included in the pre-trained de-occluded 3D facial information generation model: Based on the third preset feature constraint information, the implicit encoding information, and the lower half face geometry information included in the preset three-dimensional facial geometry information, the lower half face deformation information corresponding to the lower half face geometry information is determined; Based on the mapping association weight information, the first preset feature constraint information and the set of spatial feature point information corresponding to the preset three-dimensional facial geometry information, a spatial feature point information group corresponding to the preset three-dimensional facial geometry information is generated. Based on the implicit coding information, target weight, key geometric information of the upper half of the face included in the spatial feature point information group, and upper half of the face included in the preset three-dimensional facial geometric information, upper half of the face deformation information corresponding to the upper half of the face geometric information is generated. Based on the upper half facial deformation information and the lower half facial deformation information, the preset three-dimensional facial geometric information is modified to obtain modified three-dimensional facial geometric information. The set of spatial feature points corresponding to the modified three-dimensional facial geometry is rendered to obtain the de-occluded facial image information of the corresponding target object.

2. The method according to claim 1, wherein, The method further includes: Based on the preset model information and the three-dimensional Gaussian deformation field included in the decoder, preset three-dimensional facial geometric information is generated.

3. The method according to claim 1, wherein, The method further includes: The acquired facial image information is updated to obtain modified facial image information, wherein the facial image information represents the facial image of the target object in an occluded state; Based on the preset occlusion area information, the modified facial image information is segmented to obtain a facial image information group as a preset facial image information group, wherein the preset facial image information group includes lower half facial image information and upper half facial image information.

4. The method according to claim 1, wherein, The step of generating upper-face deformation information corresponding to the upper-face geometric information based on the implicit coding information, target weights, upper-face key geometric information included in the spatial feature point information group, and upper-face geometric information included in the preset three-dimensional facial geometric information includes: Based on the key geometric information of the upper face and the spatial feature point information of the upper face geometry, generate distance weights between the spatial feature point information of the upper face geometry and the spatial feature point information of the key geometric information of the upper face. The distance weights are normalized to obtain the target distance weights; Based on the target distance weight, the target weight, the implicit encoding information, the fourth preset feature constraint information, the upper half face geometric information, and the upper half face key geometric information, upper half face deformation information corresponding to the upper half face geometric information is generated.

5. The method according to claim 1, wherein, The step of modifying the preset three-dimensional facial geometry information based on the upper half facial deformation information and the lower half facial deformation information to obtain modified three-dimensional facial geometry information includes: Based on the fifth preset feature constraint information and the implicit encoding information, pose deformation information corresponding to the preset three-dimensional facial geometric information is generated; Based on the pose deformation information, the upper half facial deformation information, and the lower half facial deformation information, the preset three-dimensional facial geometry information is modified to obtain modified three-dimensional facial geometry information.

6. An occlusion removal device for a real-time mixed reality head-mounted display, comprising: The generation unit is configured to generate implicit encoded information based on a preset set of facial image information by using the encoder included in a pre-trained de-occluded 3D facial information generation model. The execution unit is configured to perform the following steps using a decoder included in a pre-trained de-occlusion 3D facial information generation model: determining lower half-face deformation information corresponding to the lower half-face geometry based on third preset feature constraint information, the implicit encoding information, and the lower half-face geometry included in preset 3D facial geometry; generating a set of spatial feature points corresponding to the preset 3D facial geometry based on mapping association weight information, first preset feature constraint information, and a set of spatial feature points corresponding to the preset 3D facial geometry; and generating upper half-face deformation information corresponding to the upper half-face geometry based on the implicit encoding information, target weights, key upper half-face geometry included in the spatial feature point information set, and the upper half-face geometry included in the preset 3D facial geometry. Based on the upper half facial deformation information and the lower half facial deformation information, the preset three-dimensional facial geometric information is modified to obtain modified three-dimensional facial geometric information. The set of spatial feature points corresponding to the modified three-dimensional facial geometry is rendered to obtain the de-occluded facial image information of the corresponding target object.

7. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images