An augmented reality-based object occlusion relationship confirmation method and application

Abstract

The application discloses an object occlusion relationship confirmation method and application based on augmented reality, comprising the following steps: extracting first object information of a real object in a real image; inputting the extracted first object information and second object information of a preset virtual object into an occlusion prediction network at the same time, wherein the occlusion prediction network comprises an encoder and a decoder, the first N layers of the decoder are provided with a global context structure used for aggregating global information, a detail enhancement module is arranged between the encoder and the decoder, and a loss is calculated based on an occlusion mask predicted by the occlusion prediction network and a corresponding real mask in a training process; and outputting a target mask based on the trained occlusion prediction network, so as to predict an occlusion relationship between the first object and the second object based on the target mask. The embodiment of the application is used for predicting the mutual occlusion between a real object and a virtual object, so that the effect is more realistic in an actual application scene.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and application for confirming object occlusion relationships based on augmented reality. Background Technology

[0002] To make the interaction between real and virtual objects more realistic, Battisi et al. proposed automatically locating the hand in RGB images and estimating the hand's depth information using an auxiliary infrared stereo camera for hand tracking (Battisti C, Messelodi S, Poiesi F. Seamless bare-hand interaction in mixed reality[C] / / 2018IEEE International Symposium on Mixed and Augmented Reality Adjunct(ISMAR-Adjunct).IEEE,2018:198-203.). To better correlate the hand's depth information with related pixels in the RGB image, this paper uses a distance transform algorithm to closely link the hand's depth information and pixel information. Then, the hand's depth information is compared with the rendered depth of the virtual object to determine the occlusion between the hand and the virtual object.

[0003] Battisti et al. used the LeapMotion depth sensor device to obtain hand depth information and then compared it with the rendered depth of virtual objects to determine the occlusion between the hand and the virtual objects. However, this method requires an additional depth sensor and suffers from inaccuracies in hand depth information due to angle issues and mismatches between the acquired and rendered depths, resulting in significant differences between the hand grasping virtual objects and real-world situations. Qi et al. proposed a method that uses the LeapMotion depth sensor device combined with hand tracking to predict the occlusion between the hand and virtual objects, but this method requires manual setting of the user's hand shape, making it cumbersome to use. Summary of the Invention

[0004] This application provides a method and application for confirming object occlusion relationships based on augmented reality, which is used to predict the mutual occlusion between real objects and virtual objects, making the effect more realistic in actual application scenarios.

[0005] This application provides a method for confirming object occlusion relationships based on augmented reality, including:

[0006] Extract the first object information of real objects from real images;

[0007] Based on the extracted first object information and the second object information of the preset virtual object, the occlusion prediction network is simultaneously input. The occlusion prediction network is used to predict the occlusion relationship between the first object and the second object. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are set with a global context structure to aggregate global information, and a detail enhancement module is set between the encoder and the decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding real mask.

[0008] Based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the first object and the second object.

[0009] Optionally, both the encoder and the decoder include a fusion structure of multiple convolutional layers and BN layers, with the detail enhancement module provided between some fusion structures of equal depth.

[0010] Optionally, the detail enhancement module is configured to extract detail features of the first object and the second object from any layer of the fusion structure of the first M layers of the encoder, and calculate enhanced low-level features that satisfy:

[0011] F′ l =(α-U(M)*β)*F l

[0012] Where U(M) represents upsampling the predicted confidence level M, and the predicted confidence level M is a product of the high-level features F. h F is obtained using convolution operations and softmax extraction. l F' represents the output feature of the hierarchical fusion structure corresponding to the decoder. l α represents the enhanced low-level features of the hierarchical fusion structure corresponding to the decoder, α represents the computational cardinality, and β represents the scaling factor.

[0013] Optionally, the detail enhancement module is further configured to upsample the high-level features F h With enhanced low-level features F' l They are combined and fused through convolution operations.

[0014] Optionally, the process of training the occlusion prediction network includes:

[0015] For the prediction results of the occlusion prediction network, the softmax function is used to obtain the label mapping, satisfying:

[0016]

[0017] Among them, b iLet p represent the category of any pixel i, k represent different categories, and p i,k This is the output before the softmax layer in the occlusion prediction network, representing the probability that any pixel i belongs to class k. λ is a weight-related parameter used to control the weights for each class. λ p i,k This represents the harmonic weight of pixel i belonging to category k;

[0018] Calculate category b using the Sobel operator. i and real label y i gradient direction t(b) i ),t(y i And the mean square loss within the angular domain is used as the local smoothing loss function to obtain a smooth boundary, satisfying:

[0019]

[0020] Where H×W represents the size of the target region.

[0021] Optionally, the process of training the occlusion prediction network further includes training using the following loss function:

[0022]

[0023] Among them, L o The iterative loss is represented by Ω, which represents the overlap between the hand and the virtual object. β and δ are used to control the value of Ω. i and p i Let i represent the true class and the predicted class of any pixel i, respectively.

[0024] Optionally, the process of training the occlusion prediction network further includes training using the following loss function:

[0025]

[0026] Where Ω represents the overlapping area between the hand and the virtual object, β and δ are used to control the value of Ω, and y i and p i Let i represent the true class and the predicted class of any pixel i, respectively. This represents the loss at the j-th iteration.

[0027] This application also proposes an augmented reality-based object grasping method, including the following steps:

[0028] Extract hand information from real images;

[0029] Based on the extracted hand information and the virtual object information of the preset virtual object, the information is simultaneously input into the occlusion prediction network. The occlusion prediction network is used to predict the occlusion relationship between the first object and the second object. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are set with a global context structure to aggregate global information, and a detail enhancement module is set between the encoder and the decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding real mask.

[0030] Based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the hand and the virtual object.

[0031] Object grasping is performed based on predicted occlusion relationships.

[0032] This application also proposes a terminal device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned augmented reality-based object occlusion relationship confirmation method.

[0033] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned augmented reality-based object occlusion relationship confirmation method.

[0034] This application's embodiments design a deep learning neural network model. Through a global context module and a detail enhancement module, the model can focus on global information and detailed features, enabling it to predict the mutual occlusion between real and virtual objects, resulting in more realistic effects in practical application scenarios.

[0035] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0037] Figure 1 This is a basic flowchart of the object occlusion relationship confirmation method according to an embodiment of this application;

[0038] Figure 2 The flow chart of the input-output relationship of the occlusion prediction network in this application embodiment is as follows;

[0039] Figure 3 This is an example of an occlusion prediction network structure according to an embodiment of this application;

[0040] Figure 4 This is an example of the detail enhancement module structure of the occlusion prediction network in an embodiment of this application. Detailed Implementation

[0041] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0042] This application provides a method for confirming object occlusion relationships based on augmented reality, such as... Figure 1 As shown, it includes the following steps:

[0043] This application embodiment is used to determine the occlusion relationship between real objects and virtual objects. In step S101, the first object information of the real object in the real image is extracted.

[0044] In step S102, the extracted first object information and the second object information of the preset virtual object are simultaneously input into the occlusion prediction network. For example... Figure 2 As shown, the occlusion prediction network is used to predict the occlusion relationship between the first object and the second object. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are configured with a Global Context Structure (GC) to aggregate global information, and a detail enhancement module is provided between the encoder and decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding ground truth mask. In step S103, based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the first object and the second object.

[0045] This application's embodiments design a deep learning neural network model. Through a global context module and a detail enhancement module, the model can focus on global information and detailed features, enabling it to predict the mutual occlusion between real and virtual objects, resulting in more realistic effects in practical application scenarios.

[0046] In some embodiments, both the encoder and the decoder include a fusion structure of multiple layers of convolutional layers and batch normalization (BN) layers, with the detail enhancement module disposed between fusion structures of roughly equal depth. Figure 3 As shown in the illustration, to address the limitation of convolutional operations in extracting only local features, this application introduces a Global Context (GC) structure in the first two layers of the decoder to aggregate global information. Since the features extracted by the convolutional layers often contain rich detail information, this application introduces a detail enhancement module to more clearly obtain the boundary between the hand and the virtual object. Figure 3 As shown, detail enhancement modules are set in the first four layers of the encoder and connected to the corresponding layers of the decoder, and the decoder outputs through convolutional layers + Softmax.

[0047] In some embodiments, such as Figure 4 As shown, the detail enhancement module is configured to extract detail features of the first object and the second object from any layer of the fusion structure of the first M layers of the encoder, and calculate the enhanced low-level features, satisfying:

[0048] F′ l =(α-U(M)*β)*F l

[0049] Where U(M) represents upsampling the predicted confidence level M, where the predicted confidence level M is a measure of the high-level (deep-level) features F. h F is obtained using convolution operations and softmax extraction. l F' represents the output feature of the hierarchical fusion structure corresponding to the decoder. l α represents the enhanced low-level features of the hierarchical fusion structure corresponding to the decoder, α represents the computational cardinality, and β represents the scaling factor.

[0050] In some embodiments, the detail enhancement module is further configured to upsample the high-level features F h With enhanced low-level features F' l They are combined and fused through convolution operations.

[0051] In a specific example, F' l Satisfy: F' l = (2 - U(M) * 1.5) * F l , where U(M) represents upsampling the predicted confidence M, adjusting the size of M to the size of the corresponding shallow feature map, and (2-U(M)*1.5) represents scaling the pixel value range from 0.33 to 1 to 1.5 to 0.5, and using it as a weight to represent the importance of shallow detail features.

[0052] Then, in this embodiment, the high-level feature F is... h Upsampling is performed, and the upsampled high-level features are combined with the enhanced low-level features F'. lThey are combined and finally fused together through a convolution operation.

[0053] In some embodiments, the process of training the occlusion prediction network includes:

[0054] To obtain a smoother boundary between the hand and the virtual object, this application proposes a local smoothing loss function. Specifically, for the prediction result p of the occlusion prediction network, the softmax function is used to obtain the label mapping b, satisfying:

[0055]

[0056] Among them, b i Let p represent the category of any pixel i, k represent different categories, for example, k = 1~3 means there are three different categories, namely hand, virtual object and background, p i,k This is the output before the softmax layer in the occlusion prediction network, representing the probability that any pixel i belongs to class k. λ is a weight-related parameter used to control the weights for each class. This represents the harmonic weight of pixel i belonging to category k.

[0057] Calculate category b using the Sobel operator. i and real label y i gradient direction t(b) i ),t(y i And the mean square loss within the angular domain is used as the local smoothing loss function to obtain a smooth boundary, satisfying:

[0058]

[0059] Where H×W represents the size of the target region.

[0060] Besides the smooth boundary between real and virtual objects, the overlapping portion of real and virtual objects is also crucial for predicting the occlusion between them. In some embodiments, to enable the model to better identify overlapping regions, the training process of the occlusion prediction network further includes training with the following loss function:

[0061]

[0062] Among them, L o The iterative loss is represented by Ω, which represents the overlap between the hand and the virtual object. β and δ are used to control the value of Ω. i and p i Let i represent the true class and the predicted class of any pixel i, respectively. This allows the model to focus on the overlapping area between the hand and the virtual object.

[0063] The aforementioned training method, if it focuses excessively on overlapping parts, can cause the model to lose global information. To address this issue, in some embodiments, the training process of the occlusion prediction network further includes training with the following improved loss function:

[0064]

[0065] Where Ω represents the overlapping area between the hand and the virtual object, β and δ are used to control the value of Ω, and y i and p i Let i represent the true class and the predicted class of any pixel i, respectively. Let the loss be the loss at the j-th iteration. This means that from the first iteration, the model will gradually reduce its focus on overlapping regions, allowing the model to adaptively adjust the weights of global information as training iterations increase. Thus, the loss function is designed to enable the model to gradually focus on global information.

[0066] During the training of the occlusion prediction network, the model parameters can be initialized with random noise following a zero-mean Gaussian distribution with a standard deviation of 0.1. For example, the batch size is set to 32, the optimizer is Adam, and the momentum is set to 0.9. The learning rate uses a multinomial adjustment strategy with a decay rate of 0.9. Various data augmentation methods, such as random rotation, flipping, and random sharpness, are used to expand the dataset during training.

[0067] This application proposes a method for predicting mutual occlusion between real and virtual objects, making related application scenarios more natural and realistic. This application designs a deep learning neural network model that obtains global information about hand pose through a global context module and enables the model to focus on detailed features through a detail enhancement module. This application designs a loss function for obtaining a smoother boundary between the hand and the virtual object, and a loss function for obtaining a more accurate overlapping portion of the hand and the virtual object.

[0068] This application also proposes an augmented reality-based object grasping method, including the following steps:

[0069] Extract hand information from real images;

[0070] Based on the extracted hand information and the virtual object information of the preset virtual object, the information is simultaneously input into the occlusion prediction network. The occlusion prediction network is used to predict the occlusion relationship between the first object and the second object. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are set with a global context structure to aggregate global information, and a detail enhancement module is set between the encoder and the decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding real mask.

[0071] Based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the hand and the virtual object.

[0072] Object grasping is performed based on predicted occlusion relationships.

[0073] This application proposes a method for predicting the mutual occlusion between the hand and the virtual object when the hand grasps the virtual object, making the hand grasping the virtual object more natural and realistic.

[0074] This application also proposes a terminal device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the aforementioned augmented reality-based object occlusion relationship confirmation method.

[0075] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned augmented reality-based object occlusion relationship confirmation method.

[0076] This application's embodiments design a deep learning neural network model. Through a global context module and a detail enhancement module, the model can focus on global information and detailed features, enabling it to predict the mutual occlusion between real and virtual objects, resulting in more realistic effects in practical application scenarios.

[0077] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0078] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0080] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for confirming object occlusion relationships based on augmented reality, characterized in that, include: Extract the first object information of real objects from real images; Based on the extracted first object information and the second object information of the preset virtual object, the occlusion prediction network is simultaneously input. The occlusion prediction network is used to predict the occlusion relationship between the first object and the second object. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are set with a global context structure to aggregate global information, and a detail enhancement module is set between the encoder and the decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding real mask. Based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the first object and the second object. Both the encoder and the decoder include a fusion structure of multiple layers of convolutional layers and BN layers, and the detail enhancement module is set between some fusion structures of equal depth. The detail enhancement module is configured to extract detail features of the first object and the second object from any layer of the fusion structure of the first L layers of the encoder, and calculate the enhanced low-level features, satisfying: in, This indicates that the predicted confidence level M is upsampled, where the predicted confidence level M is a high-level feature. Obtained using convolution operations and softmax extraction. This represents the output features of the hierarchical fusion structure corresponding to the decoder. This represents the enhanced low-level features of the hierarchical fusion structure corresponding to the decoder. Indicates the calculation base. Indicates the scaling factor; The process of training the occlusion prediction network includes: For the prediction results of the occlusion prediction network, the softmax function is used to obtain the label mapping, satisfying: in, Represents any pixel Category To represent different categories, It is the output before the softmax layer in the occlusion prediction network, representing any pixel. Category The probability, These are weight-related parameters used to control the weights for each category. Represents pixels i Category k Harmonic weights; Calculate the category using the Sobel operator. and real labels gradient direction Furthermore, the mean square loss within the angular domain is used as the local smoothing loss function to obtain a smooth boundary, satisfying: in, Indicates the size of the target area.

2. The object occlusion relationship confirmation method based on augmented reality as described in claim 1, characterized in that, The detail enhancement module is also configured to upsample high-level features. With enhanced low-level features They are combined and fused through convolution operations.

3. The object occlusion relationship confirmation method based on augmented reality as described in claim 1, characterized in that, The process of training the occlusion prediction network also includes training using the following loss function: in, Represents the iteration loss. This indicates the overlapping area between the hand and the virtual object. and Used to control The value, and Representing any pixel i The true category and the predicted category.

4. The object occlusion relationship confirmation method based on augmented reality as described in claim 1, characterized in that, The process of training the occlusion prediction network also includes training using the following loss function: in, This indicates the overlapping area between the hand and the virtual object. and Used to control The value, and Representing any pixel i The true category and the predicted category, Indicates the first j Loss at the next iteration.

5. An object grasping method based on augmented reality, characterized in that, Includes the following steps: Extract hand information from real images; Based on the extracted hand information and the virtual object information of the preset virtual object, the information is simultaneously input into the occlusion prediction network. The occlusion prediction network is used to predict the occlusion relationship between the hand information and the virtual object information. The occlusion prediction network includes an encoder and a decoder. The first N layers of the decoder are configured with a global context structure to aggregate global information, and a detail enhancement module is set between the encoder and the decoder. During training, the loss is calculated based on the occlusion mask predicted by the occlusion prediction network and the corresponding real mask. Based on the trained occlusion prediction network, a target mask is output to predict the occlusion relationship between the hand and the virtual object. Object grasping is performed based on predicted occlusion relationships; Both the encoder and the decoder include a fusion structure of multiple layers of convolutional layers and BN layers, and the detail enhancement module is set between some fusion structures of equal depth. The detail enhancement module is configured to extract detailed features of the hand information and the virtual object information from any layer of the fusion structure of the first L layers of the encoder, and calculate the enhanced low-level features to satisfy: in, This indicates that the predicted confidence level M is upsampled, where the predicted confidence level M is a high-level feature. Obtained using convolution operations and softmax extraction. This represents the output features of the hierarchical fusion structure corresponding to the decoder. This represents the enhanced low-level features of the hierarchical fusion structure corresponding to the decoder. Indicates the calculation base. Indicates the scaling factor; The process of training the occlusion prediction network includes: For the prediction results of the occlusion prediction network, the softmax function is used to obtain the label mapping, satisfying: in, Represents any pixel Category To represent different categories, It is the output before the softmax layer in the occlusion prediction network, representing any pixel. Category The probability, These are weight-related parameters used to control the weights for each category. Represents pixels i Category k Harmonic weights; Calculate the category using the Sobel operator. and real labels gradient direction Furthermore, the mean square loss within the angular domain is used as the local smoothing loss function to obtain a smooth boundary, satisfying: in, Indicates the size of the target area.

6. A terminal device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the steps of the augmented reality-based object occlusion relationship confirmation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the augmented reality-based object occlusion relationship confirmation method as described in any one of claims 1 to 4.

Images