AR real-scene navigation method, device, and storage medium

Abstract

The present application discloses an AR real-scene navigation method, device, and storage medium, which relate to the field of image processing technology. The AR real-scene navigation method includes: obtaining image data collected in real time during the AR real-scene navigation process; using the backbone network in the encoder of a preset semantic segmentation model to perform feature extraction on the image data to obtain shallow feature data and deep feature data; using the void space convolution pooling pyramid in the encoder to perform feature extraction on the deep feature data to obtain first feature data; using the decoder of the preset semantic segmentation model to perform feature processing on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data; based on the set of entity masks, rendering a first set of virtual objects in the AR real scene; and providing navigation guidance based on the rendered and displayed first set of virtual objects, thereby improving the accuracy of navigation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an AR real-scene navigation method, device, and storage medium. Background Art

[0002] Currently, augmented reality (AR) technology can overlay virtual navigation information on real-world road images, allowing users to intuitively understand the route and thus reducing the difficulty of understanding navigation information. However, conventional AR real-world navigation simply overlays virtual icons and routes on the real-world image, resulting in low navigation accuracy.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide an AR real-scene navigation method, device and storage medium, aiming to solve the technical problem of low navigation accuracy.

[0005] To achieve the above objectives, this application proposes an AR real-scene navigation method, which includes:

[0006] Obtain image data collected in real time during AR real-scene navigation;

[0007] The backbone network in the encoder of the preset semantic segmentation model is used to extract features from the image data to obtain shallow feature data and deep feature data;

[0008] The deep feature data is extracted using the dilated spatial convolution pooling pyramid in the encoder to obtain the first feature data;

[0009] Using a decoder of a preset semantic segmentation model to perform feature processing on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data;

[0010] Based on the entity mask set, a first virtual object set in the AR real scene is rendered;

[0011] Navigation guidance is provided based on the first set of virtual objects displayed by rendering.

[0012] In one embodiment, the steps of extracting features from image data using a backbone network in an encoder of a preset semantic segmentation model to obtain shallow feature data and deep feature data include:

[0013] The image data is passed through the first point-by-point convolution module, the first three-by-three convolution module, the attention mechanism module, and the second point-by-point convolution module in the backbone network in sequence to obtain shallow feature data and deep feature data.

[0014] In one embodiment, the step of extracting features from deep feature data using a dilated spatial convolutional pooling pyramid in an encoder to obtain first feature data includes:

[0015] The deep feature data is processed by the dilated convolution layer, the third point-by-point convolution module, and the first global average pooling layer in the dilated spatial convolution pooling pyramid, respectively, to obtain the first feature vectors output by the dilated convolution layer, the third point-by-point convolution module, and the first global average pooling layer respectively;

[0016] Concatenate all first eigenvectors in the channel dimension to obtain a first multi-channel eigenvector;

[0017] The first multi-channel feature vector is convolved through a fourth point-by-point convolution module to obtain first feature data.

[0018] In one embodiment, the step of performing feature processing on the shallow feature data and the first feature data using a decoder of a preset semantic segmentation model to obtain a set of entity masks corresponding to the image data includes:

[0019] Passing the shallow feature data through the fifth point-by-point convolution module of the decoder of the preset semantic segmentation model to obtain a second feature vector;

[0020] Processing the first feature data through a first upsampling module to obtain upsampled first feature data;

[0021] Concatenate the second eigenvector with the upsampled first eigendata in the channel dimension to obtain a second multi-channel eigenvector;

[0022] Passing the second multi-channel feature vector sequentially through the second three-by-three convolution module and the second upsampling module to obtain a prediction result;

[0023] Based on the prediction results, a set of entity masks corresponding to the image data is obtained.

[0024] In one embodiment, the step of obtaining a set of entity masks corresponding to the image data based on the prediction result includes:

[0025] Processing the prediction result using a closed operation filter to obtain a first prediction result;

[0026] Processing the first prediction result using an open operation filter to obtain a second prediction result;

[0027] Processing the second prediction result using a Gaussian filter to obtain a post-processing prediction result;

[0028] Based on the post-processing prediction results, a set of entity masks is determined.

[0029] In one embodiment, the step of rendering a first set of virtual objects in an AR real scene based on a set of entity masks includes:

[0030] determining a user's attention threshold relative to an object in the image data;

[0031] The product of the attention threshold and the area of the image data is used as the area threshold;

[0032] If it is recognized that the user is paying attention to the first object, select the entity masks with an area greater than or equal to the area threshold from the entity mask set as the first set of objects to be rendered, and render according to the relevant information of the first set of objects to be rendered to obtain a first set of virtual objects in the AR real scene;

[0033] If it is recognized that the user is paying attention to the second object, a physical mask with an area less than the area threshold is selected from the physical mask set as the second set of objects to be rendered, and rendering is performed according to the relevant information of the second set of objects to be rendered to obtain a first set of virtual objects in the AR real scene, wherein the first object is a physical mask with a polygonal area greater than or equal to the area threshold, and the second object is a physical mask with a polygonal area less than the area threshold.

[0034] In one embodiment, before the step of extracting features from image data using a backbone network in an encoder of a preset semantic segmentation model to obtain shallow feature data and deep feature data, the method further includes:

[0035] Obtain traffic scene image dataset;

[0036] Input the traffic scene image dataset into the semantic segmentation model to obtain the prediction results of the semantic segmentation model;

[0037] Use the loss function to calculate the loss value between the predicted result and the standard preset result of the traffic scene image dataset;

[0038] If the loss value is greater than or equal to the preset loss value, the weight coefficient of the semantic segmentation model is adjusted according to the loss value to obtain the adjusted semantic segmentation model, and the process returns to execute the input of the traffic scene image dataset into the semantic segmentation model to obtain the prediction result of the semantic segmentation model; and / or,

[0039] If the loss value is less than the preset loss value, the preset image segmentation model is output.

[0040] In one embodiment, the step of providing navigation guidance based on the rendered and displayed first set of virtual objects includes:

[0041] Get user navigation information;

[0042] Determining a navigation guide identifier according to the user navigation information and the first virtual object set;

[0043] Render and display navigation guidance signs on the AR real-scene interface.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes an AR real-scene navigation device, which includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, and the computer program is configured to implement the steps of the AR real-scene navigation method as described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the AR real-scene navigation method as described above are implemented.

[0046] This application obtains image data collected in real time during AR real-scene navigation; uses the backbone network in the encoder of the preset semantic segmentation model to extract features from the image data to obtain shallow feature data and deep feature data; uses the void space convolution pooling pyramid in the encoder to extract features from the deep feature data to obtain first feature data; uses the decoder of the preset semantic segmentation model to perform feature processing on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data; based on the set of entity masks, renders a first set of virtual objects in the AR real scene; and provides navigation guidance based on the rendered and displayed first set of virtual objects. Since the preset semantic segmentation model is used to extract features from the image data, objects in the image data are segmented and rendered into the AR real scene, various objects in real dynamic scenes can be recognized in real time, thereby improving the accuracy of navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of the first embodiment of the AR real-scene navigation method provided in this application;

[0050] Figure 2 This is a schematic diagram of the backbone network architecture of the AR real-scene navigation method of this application;

[0051] Figure 3A detailed flowchart of step S30 in the first embodiment of the AR real scene navigation method of this application is provided;

[0052] Figure 4 A detailed flowchart of step S40 in the first embodiment of the AR real scene navigation method of this application is provided;

[0053] Figure 5 A detailed flowchart of step S45 in the first embodiment of the AR real scene navigation method of this application is provided;

[0054] Figure 6 A detailed flowchart of step S50 in the first embodiment of the AR real scene navigation method of this application is provided;

[0055] Figure 7 A detailed flowchart of step S60 in the first embodiment of the AR real scene navigation method of this application is provided;

[0056] Figure 8 A flowchart of the second embodiment of the AR real-scene navigation method of this application is provided;

[0057] Figure 9 This is a flowchart of the AR real-scene navigation method applied in this application;

[0058] Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the AR real-scene navigation method in the embodiment of the present application.

[0059] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Traditional 2D map navigation usually requires users to construct a correspondence between the map and the real scene in their minds. This may make it difficult for people with poor sense of direction or unfamiliar with maps to understand and follow navigation instructions. Augmented reality (AR) technology can overlay virtual navigation information on the real road picture, allowing users to intuitively understand the route and reduce the difficulty of reading maps and understanding navigation information. AR is used below to represent augmented reality technology. However, conventional AR real-scene navigation simply overlays virtual icons and routes on the real scene, and cannot recognize various objects in real dynamic scenes in real time, and the navigation accuracy is not high.

[0063] In response to the above problems, the main solution of this application is: obtaining image data collected in real time during AR real-scene navigation; using the backbone network in the encoder of the preset semantic segmentation model to extract features from the image data to obtain shallow feature data and deep feature data; using the void space convolution pooling pyramid in the encoder to extract features from the deep feature data to obtain first feature data; using the decoder of the preset semantic segmentation model to perform feature processing on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data; based on the set of entity masks, rendering a first set of virtual objects in the AR real scene; and providing navigation guidance based on the rendered and displayed first set of virtual objects.

[0064] The solution of this application uses a preset semantic segmentation model to extract features from image data, thereby segmenting objects in the image data and rendering them into AR real scenes. It can recognize various objects in real dynamic scenes in real time and improve navigation accuracy.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, an AR real-scene navigation device, etc. The following uses the AR real-scene navigation device as an example to illustrate this embodiment and the following embodiments.

[0066] It should be noted that the first point-by-point convolution module, the second point-by-point convolution module, the third point-by-point convolution module, the fourth point-by-point convolution module and the fifth point-by-point convolution module can be five different convolution modules, or they can be five convolution modules with at least one group of modules being the same; the first three-by-three convolution module and the second three-by-three convolution module can be the same or different; the first upsampling module and the second upsampling module can be the same or different.

[0067] Based on this, the embodiment of the present application provides an AR real scene navigation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the AR real-scene navigation method of this application.

[0068] In this embodiment, the AR real scene navigation method includes steps S10 to S60:

[0069] Step S10: Acquire image data collected in real time during the AR real-scene navigation process.

[0070] It should be noted that the image data may be traffic scene images collected in real time.

[0071] In this embodiment, the Android platform uses the CameraX library (a Jetpack library launched by Google to simplify camera development on Android devices) to collect traffic video streams in real time, and then obtains image data by segmenting the traffic video streams into frames, thereby achieving real-time data acquisition.

[0072] Step S20: Using the backbone network in the encoder of the preset semantic segmentation model to extract features from the image data, and obtain shallow feature data and deep feature data.

[0073] It should be noted that a semantic segmentation model is a model used in the field of computer vision to classify each pixel in an image into predefined categories. For example, in an image of a traffic scene, the semantic segmentation model can label each pixel as a different category, such as "road," "building," "pedestrian," or "vehicle," so that each part of the image has a clear semantic identity. The preset semantic segmentation model is a semantic segmentation model trained on the traffic scene image dataset of this embodiment.

[0074] Furthermore, step S20 includes step S21:

[0075] In step S21, the image data is sequentially passed through the first point-by-point convolution module, the first three-by-three convolution module, the attention mechanism module, and the second point-by-point convolution module in the backbone network to obtain shallow feature data and deep feature data.

[0076] It should be noted that Figure 2 Schematic diagram of the backbone network architecture.

[0077] The first three-by-three convolution module is a convolution module with a three-by-three convolution kernel, and the attention mechanism module can be an SE (Squeeze-and-Excitation) module, which includes a global average pooling layer, a fully connected layer, an activation function layer, and a weighted pooling layer.

[0078] Shallow feature data contains detailed information of image data, such as low-level features such as edges and textures, while deep feature data contains semantic information and high-level abstract features of the captured image data.

[0079] In this embodiment, the image data is first processed through a first point-by-point convolution module to change the number of channels to obtain first image data, so that the feature information of different channels is linearly combined to extract more representative features; then a convolution operation is performed on a local area of the first image data through a first three by three convolution module to obtain second image data. Compared with point-by-point convolution, the three by three convolution kernel has a larger receptive field and can obtain richer local context information, which is conducive to extracting more complex visual features.

[0080] The attention mechanism module then learns the importance weights for each feature channel or spatial position in the second image data, recalibrating the features to produce the third image data. Specifically, the global average pooling layer in the attention mechanism compresses the two-dimensional feature map of each channel in the second image data into a real number. This real number represents the global information of the corresponding channel feature map, enabling the model to globally capture the dependencies between channel feature maps. A multi-layer perceptron (MLP) consisting of two fully connected layers then performs a nonlinear transformation on the features obtained by the global average pooling layer, learning the interdependencies between channels and generating a weight coefficient for each channel. The weight coefficients output by the fully connected layer are then normalized and compressed to the interval [0, 1] to obtain a per-channel weight value. These weight values are used to weight the channels of the second image data. The weight vector composed of these normalized weight coefficients is multiplied by each channel of the second image data to weight the feature map channels, producing the third image data. This emphasizes the features of important channels and suppresses those of unimportant channels, enabling the model to focus on specific tasks in a biased manner.

[0081] The third image data is passed through the second point-by-point convolution module, and the features are again adjusted in channel dimension and fused to obtain the fourth image data. Since the features of the third image data have already contained rich local and global information after being processed by the previous module, the second point-by-point convolution module can further integrate these features and adjust the number of channels according to task requirements to obtain the fourth image data.

[0082] The shallow feature data can be the second image data, or it can be the data obtained by splicing or weighted summing the first image data and the second image data, which can capture the local spatial structure of the image and retain the detailed information of the image data; the deep feature data can be the fourth image data, or it can be the data obtained by splicing or weighted summing the first image data, the second image data, the third image data and the fourth image data, and then aggregating the features through a convolution module or a pooling layer, so as to integrate the feature advantages of different levels, so that the deep feature data can more comprehensively capture the semantic information and high-level abstract features of the image data.

[0083] Step S30: Use the dilated spatial convolution pooling pyramid in the encoder to extract features from the deep feature data to obtain first feature data.

[0084] It should be noted that Atrous Spatial Pyramid Pooling (ASPP) is an important module for computer vision tasks such as semantic segmentation, designed to effectively capture multi-scale contextual information. Atrous Spatial Pyramid Pooling uses atrous convolutions with different dilation rates to create multiple branches with different receptive fields, thereby sampling image features at different scales. Based on standard convolution, atrous convolution inserts holes between convolution kernel elements, allowing the convolution kernel to expand its receptive field without increasing parameters or computational complexity.

[0085] Further, refer to Figure 3 , step S30 includes steps S31 to S33:

[0086] In step S31, the deep feature data is processed by the dilated convolution layer, the third point-by-point convolution module, and the first global average pooling layer in the dilated spatial convolution pooling pyramid to obtain the first feature vectors output by the dilated convolution layer, the third point-by-point convolution module, and the first global average pooling layer respectively.

[0087] It should be noted that the first global average pooling layer is a pre-set global average pooling layer, which is intended to capture the global information of the feature image.

[0088] In this embodiment, the dilated spatial convolution pooling pyramid includes three three-by-three dilated convolution layers with different dilation rates, and the dilation rates of these three dilated convolution layers are set to 6, 12, and 18, respectively, to cover features of different scale ranges. The receptive field of the dilated convolution layer with a smaller dilation rate is relatively small, which can capture detailed features in the image and is suitable for identifying smaller objects or detailed parts of objects; while the receptive field of the convolution layer with a larger dilation rate is larger, and it focuses more on capturing global contextual information, which helps to understand the overall structure of larger objects or scenes in the image. By using three-by-three dilated convolution layers with dilation rates of 6, 12, and 18 on the deep feature data, the first eigenvector corresponding to each dilated convolution layer is obtained. At the same time, the first global average pooling layer is used on the deep feature data to perform average pooling on the entire feature map in the spatial dimension, obtaining a feature vector containing global context information. The number of channels of this feature vector is then adjusted to the same as the number of channels of the first feature vector output by the dilated convolution layer through point-by-point convolution, and the spatial dimension of this feature vector is adjusted to the same as the spatial dimension of the first feature vector output by the dilated convolution layer through upsampling, obtaining the first feature vector output by the first global average pooling layer. In addition, the third point-by-point convolution module is used in parallel to process the deep feature data to obtain the first feature vector output by the third point-by-point convolution module. Thus, multiple first feature vectors are obtained by parallel processing of the deep feature data by three three-by-three dilated convolution layers with dilation rates of 6, 12, and 18, the third point-by-point convolution module, and the first global average pooling layer.

[0089] Step S32: concatenate all first eigenvectors in the channel dimension to obtain a first multi-channel eigenvector.

[0090] In this embodiment, the first multi-channel feature vector is obtained by concatenating the first feature vectors obtained in step S31 along the channel dimension. This allows the multi-scale local information captured by the dilated convolution layer and the global information provided by the first global average pooling layer to complement each other, allowing the model to focus on both local detail features and the overall scene structure, thereby performing more accurate semantic segmentation.

[0091] Step S33: Perform convolution processing on the first multi-channel feature vector through a fourth point-by-point convolution module to obtain first feature data.

[0092] In this embodiment, the fourth point-by-point convolution module is used to perform feature fusion and dimensionality adjustment on the first multi-channel feature vector to obtain first feature data. The first feature data integrates multi-scale context information and provides a rich and valuable feature representation for the subsequent semantic segmentation task.

[0093] Step S40: Using a decoder of a preset semantic segmentation model, feature processing is performed on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data.

[0094] It should be noted that the entity mask is the result of segmenting the objects in the image data by the preset semantic segmentation model. Each entity mask corresponds to an object in the image data, and all entity masks are combined into an entity mask set.

[0095] Further, refer to Figure 4 , step S40 includes steps S41 to S45:

[0096] Step S41: Pass the shallow feature data through the fifth point-by-point convolution module of the decoder of the preset semantic segmentation model to obtain a second feature vector.

[0097] In this embodiment, the fifth point-by-point convolution module is used to perform feature fusion and dimensionality adjustment on the shallow feature data to obtain the second feature data. This ensures that the second feature data has the same number of channels as the subsequently spliced data, thus ensuring that the two can be smoothly spliced in terms of the channel dimension. Furthermore, by weighting the features of each channel of the shallow feature data, the feature channels that are more important to the current task are highlighted, making the fused features more conducive to subsequent upsampling and final prediction. Furthermore, the overall number of model parameters and computational complexity can be reduced without significantly affecting model performance.

[0098] Step S42: Process the first feature data through a first upsampling module to obtain upsampled first feature data.

[0099] Step S43: Concatenate the second feature vector and the upsampled first feature data in the channel dimension to obtain a second multi-channel feature vector.

[0100] It should be noted that in image processing and deep learning, upsampling is an operation that converts a low-resolution image or feature map to a high-resolution one. Upsampling methods include bilinear interpolation and nearest neighbor interpolation.

[0101] In this embodiment, the first upsampling module uses bilinear interpolation to upsample the first feature data by a factor of four to obtain upsampled first feature data. Referring to step S32, the upsampled first feature data is concatenated with the second feature vector along the channel dimension to obtain a second multi-channel feature vector. This enables the preset semantic segmentation model to combine shallow feature data and deep feature data for feature extraction, thereby performing more accurate semantic segmentation.

[0102] Step S44: Pass the second multi-channel feature vector through the second three-by-three convolution module and the second upsampling module in sequence to obtain a prediction result.

[0103] It should be noted that the prediction result is a two-dimensional matrix with the same spatial dimensions as the input image. Each element in the matrix corresponds to a pixel in the input image, and the value of the matrix element represents the category label to which the pixel belongs. In this embodiment, the input image is image data collected in real time during AR real-scene navigation. For example, in a binary classification task, the value corresponding to the object in the physical mask is 1, and the value corresponding to the background in the physical mask is 0. In a multi-classification task, such as classifying an image into multiple categories such as roads, buildings, vehicles, and pedestrians, different categories have different corresponding values in the physical mask.

[0104] Step S45: Based on the prediction result, a set of entity masks corresponding to the image data is obtained.

[0105] In this embodiment, the second multi-channel feature vector is convolved through a second three-by-three convolution module to achieve further optimization and adjustment of the spliced second multi-channel feature vector. This can fine-tune the receptive field of the model without significantly increasing the amount of computation, so that the model can better handle objects of various scales, enhance the adaptability and robustness of the model to objects of different sizes, and improve the generalization ability of the model in complex scenarios. The second upsampling module is then used to amplify the feature vector after convolution processing by the second three-by-three convolution module by four times using bilinear interpolation upsampling to obtain the prediction result of the image data. Then, based on the multiple connected pixel areas, prediction categories and confidence corresponding to the prediction categories on the prediction results, the connected pixel areas with confidence less than the preset confidence are filtered out, and the connected pixel areas corresponding to the confidence greater than or equal to the preset confidence are taken as the entity mask. The set of multiple entity masks is the entity mask set. The upsampling operation makes the data size output by the model consistent with the image data size input by the model. In addition, the second three-by-three convolution module captures the local features of the second multi-channel feature vector. Upsampling can map the local features to a larger spatial range, which helps to restore the boundaries and details of objects in the image and improve the segmentation accuracy.

[0106] Further, refer to Figure 5 , step S45 includes steps S451 to S454:

[0107] Step S451: Use a closed operation filter to process the prediction result to obtain a first prediction result.

[0108] Step S452: Use an open operation filter to process the first prediction result to obtain a second prediction result.

[0109] Step S453: Use a Gaussian filter to process the second prediction result to obtain a post-processing prediction result.

[0110] Step S454: determining a set of entity masks based on the post-processing prediction results.

[0111] In this embodiment, when the image data contains complex texture or noise, the pixel distribution on the physical mask corresponding to larger objects in the image data may become discontinuous. Therefore, a closed operation filter is first used to sequentially perform erosion and dilation operations on the prediction result to fill small holes in the prediction result, making the pixel distribution corresponding to the larger object continuous, and changing the morphological structure of the pixel distribution corresponding to the object in the prediction result to obtain the first prediction result.

[0112] Then, the dilation and erosion operations are sequentially performed on the first prediction result using an open operation filter to remove small noise points and burrs in the first prediction result to obtain a second prediction result.

[0113] Finally, a Gaussian filter is applied to smooth the second prediction result to obtain a post-processed prediction result. Each connected pixel region in the post-processed prediction result corresponds to a physical mask of an object. Furthermore, a set of physical masks is obtained by segmenting each connected pixel region in the post-processed prediction result. This further smoothes the resulting physical mask based on the adjustments made by morphological operations such as opening and closing filters, while blurring any hard edges that may have been generated by these morphological operations. This allows for complete segmentation of larger objects in the image data, improving the accuracy of object segmentation in the image data.

[0114] Step S50: Rendering a first virtual object set in the AR real scene based on the entity mask set.

[0115] It should be noted that each pixel value in the physical mask represents the category to which the physical mask belongs. For example, to identify people and vehicles in a scene, the pixel areas of the physical mask corresponding to people and vehicles have specific values. The first virtual object set is an AR real-scene display of the object set corresponding to the physical mask set.

[0116] In this embodiment, in the recognition scene of people and vehicles, the AR development framework is used to initialize the AR real scene, obtain the image data of the device camera and the posture information of the device, such as position and direction, to determine the position and display mode of the people and vehicles in the real scene. The pixel coordinates of the physical mask corresponding to the people and vehicles are mapped to the three-dimensional space coordinates of the AR real scene. According to the category represented by the pixel area in the physical mask, the corresponding virtual object is created, and the corresponding virtual object is created in the AR real scene for the physical mask of the person, and the corresponding virtual object is created in the AR real scene for the physical mask of the vehicle. Then, using the rendering function provided by the AR development framework, the virtual objects corresponding to the people and vehicles are rendered to the corresponding three-dimensional space coordinates in the AR real scene to obtain a first virtual object set, that is, a virtual object set of people and vehicles, thereby realizing the AR display of objects in the real image data.

[0117] In another implementation of this embodiment, by receiving user selection information for objects of a specific category, selecting a physical mask of a corresponding category that meets the selection information from a physical mask set, and rendering a virtual object on an AR real scene.

[0118] Further, refer to Figure 6 , step S50 includes steps S51 to S54:

[0119] Step S51 : confirming the user's attention threshold relative to the object in the image data.

[0120] It should be noted that the attention threshold is a value greater than 0 and less than 1 set by the user, which can filter out objects that meet the user's expectations for AR real scene rendering.

[0121] In step S52, the product of the attention threshold and the area of the image data is used as the area threshold.

[0122] In step S53, if it is recognized that the user is paying attention to the first object, a physical mask with an area greater than or equal to the area threshold is selected from the physical mask set as the first set of objects to be rendered, and rendering is performed according to the relevant information of the first set of objects to be rendered to obtain a first set of virtual objects in the AR real scene.

[0123] In step S54, if it is recognized that the user is paying attention to the second object, a physical mask with an area less than the area threshold is selected from the physical mask set as the second set of objects to be rendered, and rendering is performed according to the relevant information of the second set of objects to be rendered to obtain a first set of virtual objects in the AR real scene, wherein the first object is a physical mask with a polygonal area greater than or equal to the area threshold, and the second object is a physical mask with a polygonal area less than the area threshold.

[0124] It should be noted that the first virtual object set contains no less than one piece of data.

[0125] In this embodiment, there is a car and a person in the image data. The attention threshold is received through the user terminal setting interface. The area threshold is obtained by calculating the product of the attention threshold and the area of the image data. The polygonal area of the pixel area of the physical mask corresponding to the vehicle is greater than the area threshold, and it belongs to the first object. The polygonal area of the pixel area of the physical mask corresponding to the person is less than the area threshold, and it belongs to the second object. When the user terminal setting interface receives the user's attention to the first object, refer to the method in step S50, use the pixel area of the physical mask corresponding to the vehicle to complete the rendering of the corresponding virtual object in the AR real scene, and do not render the physical mask corresponding to the person on the AR real scene, to obtain a first virtual object set, so as to achieve selective attention to objects with physical masks and meet the user's demand selection.

[0126] In another implementation of this embodiment, there is a car and a person in the image data, and the attention threshold is received through the user terminal setting interface. The area threshold is obtained by calculating the product of the attention threshold and the area of the image data. The polygonal area of the pixel area of the physical mask corresponding to the vehicle is greater than the area threshold, and it belongs to the first object. The polygonal area of the pixel area of the physical mask corresponding to the person is less than the area threshold, and it belongs to the second object. When the user terminal setting interface receives the user's attention to the second object, refer to the method in step S50, use the pixel area of the physical mask corresponding to the person to complete the rendering of the corresponding virtual object in the AR real scene, and do not render the physical mask corresponding to the vehicle on the AR real scene, to obtain a first virtual object set, so as to achieve selective attention to objects with physical masks and meet the user's demand selection.

[0127] Step S60: providing navigation guidance based on the rendered and displayed first virtual object set.

[0128] Further, refer to Figure 7 , step S60 includes steps S61-S62:

[0129] Step S61: Obtain user navigation information.

[0130] It should be noted that user navigation information is navigation route information selected by the user, including a starting point, an end point, and a route track from the starting point to the end point.

[0131] Step S62: determining a navigation guide mark according to the user navigation information and the first virtual object set, and rendering and displaying the navigation guide mark on the AR real scene interface.

[0132] It should be noted that the navigation guidance mark is a virtual mark information displayed on the AR real scene to provide navigation guidance information to the user, such as arrows, signs, positioning points, etc.

[0133] In this embodiment, one or more virtual objects are selected from the first virtual object set as navigation targets, and the map information in the AR real scene or the scene map constructed in real time by the simultaneous localization and mapping (SLAM) technology is used to receive user navigation information through the user front-end interface, so as to plan a path from the user's current location to the navigation target, and then present navigation guidance to the user through the AR interface, such as displaying guidance signs such as arrows and signs on virtual objects, or drawing a navigation path in the user's perspective. At the same time, voice prompts can also be combined to inform the user of information such as the direction and distance of travel, so as to help the user reach the navigation target smoothly and improve the accuracy of navigation.

[0134] This embodiment is implemented based on the first embodiment. In this embodiment, before step S20 of the AR real scene navigation method, refer to Figure 8 , further comprising steps B10 to B50:

[0135] Step B10: Acquire a traffic scene image dataset.

[0136] In this embodiment, polygon-annotated traffic scene image data is obtained to create a traffic scene image dataset. This dataset contains information such as the original image, mask, and category. The mask shape corresponds to the polygon shape of the annotated polygons. Furthermore, before polygon annotation, data enhancement can be performed on the traffic scene image data. This can include brightness changes, noise addition, random point addition, translation, and flipping to make the image more random and improve the model's generalization capabilities. Simultaneously, model hyperparameters are determined through a configuration table.

[0137] Step B20: Input the traffic scene image dataset into the semantic segmentation model to obtain the prediction result of the semantic segmentation model.

[0138] Step B30: using a loss function to calculate a loss value between the prediction result and a standard preset result of the traffic scene image dataset.

[0139] It should be noted that the standard preset result is the data obtained after the traffic scene image dataset is converted into the data format corresponding to the prediction result, which is used as the evaluation standard for the prediction result.

[0140] The loss function includes at least one of a cross entropy loss, a regularization loss, and a Dice loss (a type of loss function).

[0141] Step B40: If the loss value is greater than or equal to the preset loss value, the weight coefficient of the semantic segmentation model is adjusted according to the loss value to obtain the adjusted semantic segmentation model, and the execution is returned to input the traffic scene image dataset into the semantic segmentation model to obtain the prediction result of the semantic segmentation model.

[0142] Step B50, and / or, if the loss value is less than a preset loss value, output a preset image segmentation model.

[0143] In this embodiment, the loss function includes cross-entropy loss, regularization loss, and Dice loss. A traffic scene image dataset is input into a semantic segmentation model to obtain a prediction result output by the semantic segmentation model. The data in the traffic scene image dataset is formatted according to the data format of the prediction result to obtain a standard preset result. A loss function is then used to calculate the loss value between the prediction result and the standard preset result. If the loss value is greater than or equal to the preset loss value, the optimizer adjusts the weight coefficient of the semantic segmentation model based on the loss value calculated by the model. The above steps are repeated until the loss value is less than the preset loss value, completing model training and obtaining a preset semantic segmentation model, thereby providing a model foundation for achieving high-precision image object segmentation.

[0144] In another implementation of this embodiment, the traffic scene image dataset is input into the semantic segmentation model to obtain the prediction result output by the semantic segmentation model. The data in the traffic scene image dataset is formatted according to the data format of the prediction result to obtain the standard preset result. The loss function is then used to calculate the loss value between the prediction result and the standard preset result. The loss value obtained by comparison is less than the preset loss value, the model training is completed, and the preset semantic segmentation model is obtained, thereby providing a model basis for achieving high-precision image object segmentation.

[0145] For example, in order to help understand the implementation process of the AR real scene navigation method obtained by combining this embodiment with the above embodiment 1, please refer to Figure 9 , Figure 9 A brief flowchart of an AR real-scene navigation method is provided, specifically:

[0146] The encoder's backbone network receives input image data, performs feature extraction on the image data, obtains deep feature data and shallow feature data, applies the third point-by-point convolution module, the first global average pooling layer, and three hole convolution layers to process the deep feature data respectively, obtains multiple first feature vectors, splices these first feature vectors in the channel dimension to obtain a first multi-channel feature vector, and convolves the first multi-channel feature vector through the fourth point-by-point convolution module to obtain the first feature data; the decoder receives shallow feature data from the backbone network, extracts features on the shallow feature data through the fifth point-by-point convolution module, obtains a second feature vector, splices the second feature vector with the first multi-channel feature vector upsampled by the first upsampling module in the channel dimension to obtain a second multi-channel feature vector, and processes the second multi-channel feature vector in sequence using the second three-by-three convolution module and the second upsampling module to obtain the prediction result of the preset semantic segmentation model.

[0147] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the AR real-scene navigation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0148] The present application provides an AR real-scene navigation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AR real-scene navigation method in the above-mentioned embodiment one.

[0149] Reference below Figure 10 , which shows a schematic structural diagram of an AR real-scene navigation device suitable for implementing an embodiment of the present application. The AR real-scene navigation device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The AR real-scene navigation device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0150] like Figure 10As shown, the AR real scene navigation device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the AR real scene navigation device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the AR real-scene navigation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an AR real-scene navigation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0151] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0152] The AR real-scene navigation device provided in this application adopts the AR real-scene navigation method in the above-mentioned embodiment to solve the technical problem of low navigation accuracy. Compared with the existing technology, the beneficial effects of the AR real-scene navigation device provided in this application are the same as the beneficial effects of the AR real-scene navigation method provided in the above-mentioned embodiment, and the other technical features of the AR real-scene navigation device are the same as those disclosed in the method of the previous embodiment, which will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0155] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the AR real-scene navigation method in the above-mentioned embodiment.

[0156] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0157] The computer-readable storage medium may be included in the AR real-scene navigation device; or it may exist independently without being assembled into the AR real-scene navigation device.

[0158] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the AR real-scene navigation device, the AR real-scene navigation device: obtains image data collected in real time during the AR real-scene navigation process; uses the backbone network in the encoder of the preset semantic segmentation model to extract features of the image data to obtain shallow feature data and deep feature data; uses the void space convolution pooling pyramid in the encoder to extract features of the deep feature data to obtain first feature data; uses the decoder of the preset semantic segmentation model to perform feature processing on the shallow feature data and the first feature data to obtain a set of entity masks corresponding to the image data; based on the set of entity masks, renders a first set of virtual objects in the AR real scene; and provides navigation guidance based on the rendered and displayed first set of virtual objects.

[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0161] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0162] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned AR real-scene navigation method, thereby resolving the technical issue of low navigation accuracy. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the AR real-scene navigation method provided in the aforementioned embodiments, and are not further elaborated here.

[0163] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An AR real scene navigation method, characterized in that: The AR real scene navigation method includes: Obtain image data collected in real time during AR real-scene navigation; Using a backbone network in an encoder of a preset semantic segmentation model to perform feature extraction on the image data to obtain shallow feature data and deep feature data; Performing feature extraction on the deep feature data using a dilated spatial convolutional pooling pyramid in the encoder to obtain first feature data; Performing feature processing on the shallow feature data and the first feature data using a decoder of the preset semantic segmentation model to obtain a set of entity masks corresponding to the image data; determining a user's attention threshold relative to an object in the image data; The product of the attention threshold and the area of the image data is used as the area threshold; If it is recognized that the user is paying attention to the first object, select the entity masks with an area greater than or equal to the area threshold from the entity mask set as the first set of objects to be rendered, and render according to the relevant information of the first set of objects to be rendered to obtain a first set of virtual objects in the AR real scene; If it is recognized that the user is paying attention to the second object, a physical mask with an area smaller than the area threshold is selected from the physical mask set as a second set of objects to be rendered, and rendering is performed according to relevant information of the second set of objects to be rendered to obtain a first set of virtual objects in the AR real scene, wherein the first object is a physical mask with a polygonal area greater than or equal to the area threshold, the second object is a physical mask with a polygonal area smaller than the area threshold, and the first virtual object set is an AR real scene display of the object set corresponding to the physical mask set; Navigation guidance is provided based on the first set of virtual objects displayed by rendering.

2. The AR real scene navigation method according to claim 1, wherein: The step of extracting features from the image data using the backbone network in the encoder of the preset semantic segmentation model to obtain shallow feature data and deep feature data includes: The image data is sequentially passed through the first point-by-point convolution module, the first three-by-three convolution module, the attention mechanism module, and the second point-by-point convolution module in the backbone network to obtain the shallow feature data and the deep feature data.

3. The AR real scene navigation method according to claim 2, wherein: The step of extracting features from the deep feature data using the dilated spatial convolutional pooling pyramid in the encoder to obtain first feature data includes: The deep feature data is processed by the dilated convolution layer, the third point-by-point convolution module, and the first global flat pooling layer in the dilated spatial convolution pooling pyramid, respectively, to obtain first feature vectors output by the dilated convolution layer, the third point-by-point convolution module, and the first global flat pooling layer respectively; Concatenating all the first feature vectors in the channel dimension to obtain a first multi-channel feature vector; The first multi-channel feature vector is convolved through a fourth point-by-point convolution module to obtain the first feature data.

4. The AR real scene navigation method according to claim 1 or 3, characterized in that: The step of performing feature processing on the shallow feature data and the first feature data by using the decoder of the preset semantic segmentation model to obtain a set of entity masks corresponding to the image data includes: Passing the shallow feature data through a fifth point-by-point convolution module of a decoder of the preset semantic segmentation model to obtain a second feature vector; Processing the first feature data through a first upsampling module to obtain upsampled first feature data; Concatenate the second feature vector and the upsampled first feature data in the channel dimension to obtain a second multi-channel feature vector; Passing the second multi-channel feature vector sequentially through a second three-by-three convolution module and a second upsampling module to obtain a prediction result; Based on the prediction result, a set of entity masks corresponding to the image data is obtained.

5. The AR real scene navigation method according to claim 4, wherein: The step of obtaining a set of entity masks corresponding to the image data based on the prediction result includes: Processing the prediction result using a closed operation filter to obtain a first prediction result; Processing the first prediction result using an open operation filter to obtain a second prediction result; Processing the second prediction result using a Gaussian filter to obtain a post-processing prediction result; The entity mask set is determined based on the post-processing prediction result.

6. The AR real scene navigation method according to claim 1, wherein: Before the step of extracting features from the image data using the backbone network in the encoder of the preset semantic segmentation model to obtain shallow feature data and deep feature data, the method further includes: Obtain traffic scene image dataset; Inputting the traffic scene image dataset into a semantic segmentation model to obtain a prediction result of the semantic segmentation model; Calculate the loss value between the prediction result and the standard preset result of the traffic scene image dataset using a loss function; If the loss value is greater than or equal to a preset loss value, adjusting the weight coefficient of the semantic segmentation model according to the loss value to obtain an adjusted semantic segmentation model, and returning to execute the step of inputting the traffic scene image dataset into the semantic segmentation model to obtain a prediction result of the semantic segmentation model; and / or, If the loss value is less than the preset loss value, the preset image segmentation model is output.

7. The AR real scene navigation method according to claim 1, wherein: The step of providing navigation guidance based on the first set of rendered virtual objects includes: Get user navigation information; determining a navigation guide identifier according to the user navigation information and the first set of virtual objects; The navigation guide mark is rendered and displayed on the AR real scene interface.

8. An AR real-scene navigation device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the AR real-scene navigation method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the AR real-scene navigation method according to any one of claims 1 to 7 are implemented.

Images