Methods for determining planar representation and methods and devices for displaying augmented reality information

By segmenting and estimating the depth of multimedia information, three-dimensional planar information is determined, solving the problem of discrepancies between virtual objects and real scenes in AR displays and achieving a more realistic augmented reality display effect.

CN108629800BActive Publication Date: 2026-11-17BEIJING SAMSUNG TELECOM R&D CENT +1
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN201710853701.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-03-20
Filing Date
2017-09-15
Publication Date
2026-11-17
Estimated Expiration
2037-09-15

AI Technical Summary

Technical Problem

Current AR technology displays virtual content, but the virtual objects do not match the real scene, resulting in unrealistic display results and a poor user experience.

Method used

By performing region segmentation and depth estimation on multimedia information, three-dimensional planar information is determined. Deep learning networks are used for planar classification and semantic recognition. Conditional random fields are then combined to correct the planar information, thereby improving the accuracy of planar determination.

Benefits of technology

It improves the realism of augmented reality displays and enhances the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN108629800B_ABST
    Figure CN108629800B_ABST
Patent Text Reader

Abstract

This invention provides a method for determining a plane and a method and apparatus for displaying augmented reality (AR) information. The method includes: performing region segmentation and depth estimation on multimedia information; then determining the three-dimensional plane information of the multimedia information based on the region segmentation and depth estimation results; and finally displaying AR information based on the corresponding three-dimensional plane information. The plane determination method and the method and apparatus for displaying AR information provided by this invention can be used to add virtual display information to a three-dimensional plane, improving the realism of the AR display results and enhancing the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimedia technology, and more specifically, to a method for determining a plane and a method and apparatus for displaying augmented reality information. Background Technology

[0002] Augmented Reality (AR) technology overlays virtual content onto real-world scenes, providing users with a sensory experience that surpasses reality. Users can perceive a scene where real objects and virtual content coexist. AR technology can be applied in fields such as home design, travel translation, shopping, gaming, navigation, and education.

[0003] In existing AR implementation methods, virtual content is typically placed directly into multimedia information corresponding to a real-world scene to obtain the AR display result. However, the realism of the AR display result obtained using existing methods is poor, and the user experience needs improvement. Figure 1 As shown, when a virtual object (such as an animal) is placed within multimedia information corresponding to a real-world scene, the virtual object appears to float in the air in the AR display, which does not match reality; for example... Figure 2 As shown, after virtual objects are placed within multimedia information corresponding to a real-world scene, they are attached to an unreasonable plane (such as a vertical wall); for example... Figure 3 As shown, the displayed virtual navigation route extends directly into the air; as Figure 4 As shown, the displayed virtual navigation route passes directly through the obstacles.

[0004] In summary, existing AR implementation methods will result in unrealistic AR display results and a poor user experience. Summary of the Invention

[0005] To overcome or at least partially solve the aforementioned technical problems, the following technical solutions are proposed:

[0006] Embodiments of the present invention provide a method for plane determination, comprising:

[0007] Perform region segmentation and depth estimation on multimedia information;

[0008] Based on the region segmentation results and depth estimation results, the three-dimensional planar information of the multimedia information is determined.

[0009] Embodiments of the present invention provide a plane-determining apparatus, comprising:

[0010] The processing module is used for region segmentation and depth estimation of multimedia information;

[0011] The first determining module is used to determine the three-dimensional planar information of the multimedia information based on the region segmentation result and depth estimation result obtained by the processing module.

[0012] In an embodiment of the present invention, a method for displaying augmented reality information is proposed, comprising:

[0013] Determine the three-dimensional planar information corresponding to the multimedia information;

[0014] Augmented reality display information is displayed based on the three-dimensional planar information corresponding to the multimedia information.

[0015] An embodiment of the present invention provides a display device for displaying augmented reality information, comprising:

[0016] The second determining module is used to determine the three-dimensional planar information corresponding to the multimedia information;

[0017] The display module is used to display augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information determined by the second determining module.

[0018] This invention provides a method for determining a plane and a method and apparatus for displaying augmented reality information. By performing region segmentation and depth estimation on multimedia information, this invention determines the three-dimensional plane of the multimedia information. Compared with the existing two-dimensional plane determination method, the determined three-dimensional plane is more accurate. Based on the determined three-dimensional plane information, augmented reality information is displayed, which can improve the realism of the augmented reality display results and thus enhance the user experience.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 This is a schematic diagram illustrating the display result of augmented reality information using existing technology.

[0022] Figure 2 This is a schematic diagram illustrating the display result of augmented reality information using another existing technology.

[0023] Figure 3 This is a schematic diagram illustrating the display result of augmented reality information using another existing technology.

[0024] Figure 4 This is a schematic diagram illustrating the display result of augmented reality information using another existing technology.

[0025] Figure 5 This is a flowchart illustrating a plane determination method in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the segmentation results combining semantic annotation and planar annotation in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the segmentation process based on FCN and Conditional Random Field in an embodiment of the present invention;

[0028] Figure 8 This is an integrated region segmentation framework used to simultaneously obtain planar information and semantic information in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of a segmented network architecture in an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the depth estimation network architecture in an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram showing the comparison between the prediction results during training and the actual depth of the training samples in an embodiment of the present invention.

[0032] Figure 12 This is a schematic diagram of the overall framework of the depth estimation network in an embodiment of the present invention;

[0033] Figure 13 This is a schematic diagram illustrating the use of a multi-task approach for region segmentation and depth estimation in an embodiment of the present invention;

[0034] Figure 14 This is a schematic diagram illustrating the use of a multi-task approach for region segmentation and depth estimation in an embodiment of the present invention;

[0035] Figure 15 This is a comparison diagram illustrating the advantages of using three-dimensional spatial information versus two-dimensional spatial information in planar understanding, as described in an embodiment of the present invention.

[0036] Figure 16 This is a schematic diagram illustrating the adjustment of determined three-dimensional planar information in an embodiment of the present invention;

[0037] Figure 17 This is a schematic flowchart of the method for determining a plane using hybrid methods in an embodiment of the present invention;

[0038] Figure 18 This is a schematic diagram illustrating the processing result of the method for determining a plane using a hybrid approach in an embodiment of the present invention;

[0039] Figure 19This is a flowchart illustrating the method for displaying augmented reality information in an embodiment of the present invention.

[0040] Figure 20 This is a schematic diagram illustrating the display position of automatically recommended virtual display information in an embodiment of the present invention;

[0041] Figure 21 This is a schematic diagram illustrating automatic recommendation implemented through a prior knowledge-based filtering system and a long short-term memory neural network in an embodiment of the present invention;

[0042] Figure 22 A comparison diagram of augmented reality display information generated by existing technologies and the present invention;

[0043] Figure 23 This is a schematic diagram illustrating the specific process of automatically recommending the display position of virtual display information in an embodiment of the present invention;

[0044] Figure 24 This is a schematic diagram illustrating the display position of automatically recommended virtual display information in an embodiment of the present invention;

[0045] Figure 25 This is a flowchart illustrating the planar position adjustment recommendation method in an embodiment of the present invention;

[0046] Figure 26 This is a schematic diagram illustrating the first recommended method for adjusting the planar position in an embodiment of the present invention;

[0047] Figure 27 This is a schematic diagram illustrating the second recommended method for adjusting the planar position in an embodiment of the present invention;

[0048] Figure 28 This is a schematic diagram of the method for recommending the attitude adjustment of the data acquisition device in an embodiment of the present invention;

[0049] Figure 29 This is a schematic diagram illustrating the recommended method for adjusting the attitude of the data acquisition device in an embodiment of the present invention;

[0050] Figure 30 This is a schematic diagram illustrating the process of displaying assisted driving prompts in an embodiment of the present invention;

[0051] Figure 31 This is a schematic diagram illustrating the display of assisted driving prompts in an embodiment of the present invention;

[0052] Figure 32 This is a schematic diagram of a road condition estimation method in a driving system according to an embodiment of the present invention;

[0053] Figure 32 This is a schematic diagram of the road condition estimation method in the driver assistance system according to an embodiment of the present invention;

[0054] Figure 33 This is a schematic diagram of a road condition estimation method in an assisted driving system according to an embodiment of the present invention;

[0055] Figure 34 This is a flowchart illustrating the implementation method of the AR keyboard in an embodiment of the present invention;

[0056] Figure 35 This is a schematic diagram of a multilingual AR keyboard in an embodiment of the present invention;

[0057] Figure 36 This is a schematic diagram of an encrypted password keyboard in an embodiment of the present invention;

[0058] Figure 37 This is a schematic diagram of a planar device structure in an embodiment of the present invention;

[0059] Figure 38 This is a schematic diagram of the structure of an augmented reality display device for displaying information, as described in an embodiment of the present invention. Detailed Implementation

[0060] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0061] Example 1

[0062] This invention provides a method for plane determination, such as... Figure 5 As shown, it includes the following steps:

[0063] Step 501: Perform region segmentation and depth estimation on the multimedia information.

[0064] The multimedia information in the embodiments of the present invention includes images and / or videos, etc.

[0065] Furthermore, the multimedia information in the embodiments of the present invention may be, but is not limited to, monocular multimedia information, such as multimedia information collected by a multimedia information acquisition device (such as a camera).

[0066] This invention proposes that the region segmentation result can include two-dimensional planar information. Specifically, this can be achieved, but is not limited to, by using a deep learning network to segment multimedia information to obtain the two-dimensional planar information. This includes the following step 5011 (not shown in the figure).

[0067] Step 5011: By combining the deep learning network trained with planar annotation, the multimedia information is segmented into regions to obtain the region segmentation result (two-dimensional planar information).

[0068] The aforementioned deep learning network is trained in the following way: the training samples are labeled in a plane; and the deep learning network is trained based on the labeled training samples.

[0069] This invention proposes that the region segmentation result can further include semantic information corresponding to a two-dimensional plane. Specifically, this can be achieved, but is not limited to, using a deep learning network to segment multimedia information into regions, obtaining two-dimensional plane information and its corresponding semantic information. Different deep learning networks can be used for two-dimensional plane recognition and semantic recognition, respectively obtaining the two-dimensional plane information and its corresponding semantic information.

[0070] In addition, multimedia information can be segmented into regions using the same deep learning network. That is, two-dimensional plane recognition and semantic recognition can be performed simultaneously using the same deep learning network to obtain two-dimensional plane information and the corresponding semantic information. Specifically, this includes the following steps 5012 (not marked in the figure).

[0071] Step 5012: By combining the deep learning network trained with semantic annotation and planar annotation, the multimedia information is segmented into regions to obtain the region segmentation results (two-dimensional planar information and the corresponding semantic information of the two-dimensional plane).

[0072] The aforementioned deep learning network is trained in the following way: semantic and planar annotations are performed on the training samples; and the deep learning network is trained based on the annotated training samples.

[0073] In steps 5011 and 5012 above, when training a deep learning network based on labeled training samples, the objective function and network structure of the deep learning network can be determined first; then, the deep learning network can be trained based on the labeled training samples, the objective function, and the network structure.

[0074] In this embodiment of the invention, during the training of the deep learning network, steps 5011 and 5012 can be performed on the training samples using pixel-level annotation, such as semantic labeling or planar annotation. Semantic labeling uses semantic objects as units, marking the specific location of the semantic object within the multimedia information (images are used as examples below). Pixel-level semantic labeling, on the other hand, refines the specific location to the pixel level, semantically labeling each pixel in the image used as a training sample. For example, if a vehicle is selected as a semantic object, each pixel corresponding to the vehicle in the image is labeled with the same semantic information (such as semantic attribute identifiers). In this embodiment of the invention, planar annotation uses planes as units, labeling each pixel corresponding to each plane in the image with the same planar information. For example, if an image contains N planes, each plane corresponds to several pixels, and each pixel within the same plane corresponds to the same planar information.

[0075] Since semantic and planar annotations have been performed on each pixel during the training of the deep learning network, when performing region segmentation on multimedia information through the deep learning network, the planar recognition result and semantic recognition result of each pixel can be obtained. Then, based on the planar recognition result and semantic recognition result of each pixel, the two-dimensional planar information and the corresponding semantic information of the two-dimensional plane can be determined.

[0076] The same deep learning network trained in the above manner can also be called an end-to-end trained deep learning network.

[0077] When performing planar annotation on pixels, the planar information corresponding to a pixel includes a classification identifier and / or a planar identifier. The classification identifier includes at least one of the following: a classification identifier corresponding to a plane, a classification identifier corresponding to a plane edge, and a classification identifier corresponding to a non-plane. When a pixel belongs to a plane, the planar information of that pixel includes the classification identifier corresponding to the plane and the planar identifier of the plane to which the pixel belongs; when a pixel does not belong to any plane, the planar information of that pixel includes the classification identifier corresponding to a non-plane; when a pixel is located at a plane edge, the planar information of that pixel includes the classification identifier corresponding to the plane edge and the planar identifier of the plane corresponding to the plane edge.

[0078] Unlike simple region segmentation, the region segmentation in this embodiment of the invention can be defined as a planar classification problem, rather than a segmentation problem in the prior art. Since each pixel is labeled with a classification identifier during the training of the deep learning network, the trained deep learning network can classify each pixel. Points classified as "planar" can form a connected two-dimensional plane, thus obtaining the aforementioned two-dimensional plane information.

[0079] In the above embodiments of the present invention, by combining semantic annotation and planar annotation, training samples can be endowed with dual attributes, with each pixel in the training sample corresponding to both semantic and planar annotations. For example, as Figure 6 As shown, Figure 6 (a) According to ordinary region segmentation methods, only planes in the image can be segmented, but the semantic information corresponding to each plane cannot be determined; Figure 6 (b) is a common semantic segmentation method, which can only distinguish the image content into cubes and background, but cannot specifically distinguish each plane of the cube; Figure 6 (c) is the region segmentation method used in this embodiment of the invention, which can simultaneously obtain planar information and the semantic information corresponding to the plane. Specifically: for a six-sided cube, each of the six faces corresponds to the same semantic meaning. For any pixel in the six-sided cube, semantic annotation is used to label each pixel with the same semantic attribute; and by combining planar annotation, the six planes of the cube can be further subdivided. In this embodiment of the invention, this dual attribute helps to refine multimedia information, thereby obtaining more meaningful results than simple semantic segmentation or simple region segmentation.

[0080] The following details the semantic segmentation architecture mentioned above. The deep learning network can be a fully convolutional neural network (FCN). The entire segmentation process is based on FCN and Conditional Random Field (CRF). For example... Figure 7 As shown, the input image and the semantic segmentation target are first determined. In this image, the semantic segmentation target is defined as text. A preliminary semantic segmentation result is obtained through dense prediction using a fully convolutional neural network (FCN). This preliminary semantic segmentation result corresponds to the text-containing portion of the input image. Furthermore, the FCN employs an atrous structure in both the convolutional and pyramid pooling layers to reduce downsampling and allow for multi-scale feature extraction, thereby making the semantic segmentation result more reliable and the network easier to train. Since the preliminary semantic segmentation result still contains some adhesion and blurring, a conditional random field is used, and image information such as illumination, color, and contrast are set as judgment conditions to obtain a more refined final segmentation result.

[0081] The integrated region segmentation framework (also known as a multi-task region segmentation architecture) proposed in this invention, which can simultaneously obtain planar information and semantic information, is based on the aforementioned semantic segmentation architecture. This multi-task region segmentation architecture transforms the acquisition of planar information from a traditional image processing problem into a pixel-level classification problem in deep learning, enabling planar information to be obtained through the aforementioned semantic segmentation architecture. Figure 8 As shown, the input image is first processed through a feature extraction layer composed of a fully convolutional neural network. Then, each pixel undergoes two pixel-level classifications: planar classification (determining whether the pixel belongs to a plane, a plane edge, or a non-planar element) and semantic classification (determining the semantic attribute of the pixel, such as sky, wall, or floor). The final result is a region segmentation that simultaneously contains two-dimensional planar information and corresponding semantic information.

[0082] Regarding the steps for depth estimation of multimedia information, this embodiment of the invention proposes that the accuracy of depth estimation can be improved by utilizing the temporal difference information between consecutive frames in monocular multimedia information (multimedia information acquired by a monocular camera). Specifically, depth estimation using a single frame of multimedia information only obtains spatial information, but depth estimation using multimedia information from consecutive frames not only obtains spatial information but also temporal information for each location in the multimedia information. This temporal information can be used to calibrate the depth estimation results. For example, if the difference in depth estimation results for the same location (e.g., the same pixel) between consecutive frames is too large (greater than a set threshold), the depth estimation result of the previous frame can be used to calibrate the depth estimation result of the current frame.

[0083] This invention proposes that region segmentation and depth estimation can be performed in a single-task manner. Specifically, the region segmentation result is obtained through a deep learning network (which can be called a segmentation network) corresponding to the region segmentation, and the depth estimation result is obtained through a deep learning network (which can be called a depth estimation network) corresponding to the depth estimation. Based on the obtained region segmentation result and depth estimation result, a three-dimensional plane fitting is further performed to obtain three-dimensional plane information.

[0084] In embodiments of the present invention, the segmentation network and the depth estimation network are as follows: Figure 9 as well as Figure 10 As shown, specifically, refer to Figure 9 For the segmentation network, the input image is first processed by several feature extraction layers to extract feature information, the SOFTXMAX classification layer is used to classify and segment the results, and a deconvolution layer is used to restore the original image size to obtain pixel-level segmentation results.

[0085] Reference Figure 10The first half of the depth estimation network is similar to the segmentation network, using a multi-layer residual network as the feature extraction layer. Afterwards, it gradually recovers the original feature through a series of deconvolution layers. Figure 1 The size is half, and the final result can be presented as a continuous distribution in the form of a heat map.

[0086] When training a depth estimation network, a triplet loss function (composed of absolute loss, relative loss, and fusion loss) can be used to train the network based on the true depth of the training samples. Figure 11 This is a comparison chart between the predicted depth information map during training and the actual depth of the training samples for the input image.

[0087] Specifically: In the embodiments of the present invention, the overall framework of the depth estimation network can be based on FCN, such as... Figure 12 As shown, the depth estimation network mainly consists of a 50-layer residual network (ResNet-50) in the first half of the FCN and a deconvolutional network (Deconvolutional Network) in the second half. During training, a triplet loss function is used to improve the accuracy of depth estimation and optimize the training results. First, image information is input, and then ResNet-50 extracts features from the image. This feature extraction process uses skip connections to connect several layers in the residual network together. Then, the extracted features are deconvolved through the deconvolutional network until the size is half the original image size. Skip connections are also used in the deconvolution process. The output is then upsampled to obtain a depth estimation result that corresponds one-to-one with the pixels of the original image, ultimately obtaining the depth information corresponding to each pixel in the original image. The depth estimation results can be presented as a continuous distribution in a heatmap format.

[0088] This invention proposes that, after obtaining the region segmentation result and the depth estimation result, the depth estimation result can be further corrected based on the region segmentation result; and / or, the region segmentation result can be corrected based on the depth estimation result.

[0089] The region segmentation results and depth estimation results can be obtained through the single-task method described above.

[0090] Furthermore, this embodiment of the invention proposes that a multi-task approach can be used for region segmentation and depth estimation. Through the same deep learning network, region segmentation and depth estimation are performed on multimedia information. Based on the region segmentation results, the depth estimation results are corrected, and based on the depth estimation results, the region segmentation results are corrected to obtain corrected region segmentation results and depth estimation results.

[0091] Specifically: refer to Figure 13 The system extracts features from the input image (multimedia information) through a feature extraction layer, obtaining multi-level feature information. Since the subsequent depth estimation subnetwork and region segmentation subnetwork only require feature extraction once, they do not need to be extracted separately, thus achieving shared computation between the depth estimation subnetwork and the region segmentation subnetwork. Based on the extracted multi-level feature information, the region segmentation subnetwork obtains the corresponding region segmentation result (a preliminary result), and the depth estimation subnetwork obtains the corresponding depth estimation result (a preliminary result). Through a fusion layer, the region segmentation result and the depth estimation result are fused (the depth estimation result is corrected based on the region segmentation result, and the region segmentation result is corrected based on the depth estimation result), resulting in a corrected region segmentation result and a corrected depth estimation result.

[0092] Among them, the two sub-networks (region segmentation sub-network and depth estimation sub-network) can be logical sub-networks. In the actual network architecture, the two sub-networks can be treated as a whole, i.e., a single network; or they can be treated as two separate networks.

[0093] Furthermore, the depth estimation subnetwork and the region segmentation subnetwork are trained as follows: the depth estimation subnetwork is trained using a deep learning network (which may be, but is not limited to, a residual network) as a pre-trained model; the region segmentation subnetwork is trained using the trained depth estimation subnetwork as a pre-trained model; the fusion layer of the deep learning network is trained using the trained region segmentation subnetwork as a pre-trained model; and the depth estimation subnetwork and the region segmentation subnetwork are trained using the fusion layer of the trained deep learning network as a pre-trained model.

[0094] The aforementioned depth estimation subnetwork and region segmentation subnetwork can also be trained in the following way:

[0095] A deep learning network (which may be, but is not limited to, a residual network) is used as a pre-trained model to train the region segmentation sub-network; the trained region segmentation sub-network is used as a pre-trained model to train the depth estimation sub-network; the trained depth estimation sub-network is used as a pre-trained model to train the fusion layer of the deep learning network; and the trained fusion layer of the deep learning network is used as a pre-trained model to train both the region segmentation sub-network and the depth estimation sub-network.

[0096] Preferably, when training the region segmentation subnetwork, the region segmentation subnetwork can also be trained by combining planar annotations, or planar labels and semantic annotations, in the manner mentioned above.

[0097] For example, using a residual network as a pre-trained model, the learning rate of the region segmentation sub-network is set to 0, and backpropagation is performed only on the depth estimation sub-network. The network parameters of the depth estimation sub-network are optimized based on unilateral training, i.e., the initial learning rate of the depth estimation sub-network is set to 0.01, and it is trained for about 100,000 epochs. Then, the depth estimation sub-network obtained in the previous step is used as a pre-trained model, and the learning rate of the depth estimation sub-network is set to 0. The network parameters of the segmentation sub-network are optimized unilaterally, with the same learning rate and number of training iterations. Then, the result of the previous step is used as a pre-trained model, and the learning rates of both the segmentation sub-network and the depth estimation sub-network are set to 0. The parameter information of the last fusion layer is trained. Then, the learning rate of all sub-networks is set to normal based on the result of the previous step, i.e., the overall learning rate is set to 0.001, and it is trained for 100,000 epochs to obtain the final result.

[0098] The advantages of performing depth estimation and region segmentation through multi-task processing are described below:

[0099] 1. Shared Computation: In the aforementioned single-task processing methods, both the depth estimation network and the region segmentation network are based on dense prediction using fully convolutional neural networks. Their network structures are highly similar, differing only significantly in their learning objectives. This invention leverages the high similarity in network structures found in the single-task processing methods, employing a multi-task processing approach to enable shared computation between depth estimation and region segmentation. This improves both computational speed and the accuracy of the processing results.

[0100] 2. Dependency Relationship: Depth estimation and region segmentation results are complementary and mutually constraining. For example, if the depth information of a region does not change significantly, the region is likely planar; conversely, if a region is planar, the depth information may need to maintain continuous and smooth changes. Combining these two points, unifying region segmentation and depth estimation using the same deep learning network for multi-task prediction allows for simultaneous acquisition of both depth estimation and region segmentation results. This not only shares computation and improves speed but also provides complementary constraints, thereby enhancing the reliability of the final result.

[0101] If the region segmentation result of the region segmentation subnetwork contains two-dimensional planar information and the corresponding semantic information, then when correcting the depth estimation result based on the region segmentation result, the depth estimation result can be corrected according to the two-dimensional planar information and semantic information, resulting in a more accurate depth estimation result. For example, if an image contains a window region, when estimating the depth of the window region, due to the transparent glass, the obtained depth estimation result may not be the depth of the window region, but rather the depth of objects outside the window. If the depth estimation result can be corrected according to the semantic information of the region, a more accurate depth estimation result can be obtained. Similarly, for a wall and a painting hanging on the wall, semantic information can determine that the depth information of the two is not significantly different, thus the depth estimation result can be corrected, improving the accuracy of the depth estimation result.

[0102] like Figure 14 As shown, the input image is processed by a fully convolutional network as the feature extraction layer to extract multi-level feature information. Based on the extracted multi-level feature information, a region segmentation sub-network is used to obtain the corresponding region segmentation result (this result is a preliminary result). The region segmentation result includes two-dimensional planar information and corresponding semantic information. Based on the multi-level feature information, a depth estimation sub-network is used to obtain the corresponding depth estimation result (this result is a preliminary result). Based on the depth estimation result, a cross-neighborhood conditional random field can be used to correct the region segmentation result and the depth estimation result, resulting in a corrected region segmentation result and a corrected depth estimation result for a more accurate result.

[0103] In this embodiment of the invention, a novel network structure enables simultaneous prediction of depth information and region segmentation results during shared computation. Specifically, image information is first input, and a Deep Residual Network (DRN) extracts higher-level features layer by layer. Features from multiple levels are fused through addition and other methods to obtain feature map information containing multi-level information. This multi-level feature map information is then used simultaneously for shared computation of depth estimation and region segmentation. Depth and planar information (which may also include semantic information) are then learned separately through a depth estimation sub-network and a segmentation sub-network. Finally, a single network layer fuses the two information together, simultaneously predicting both depth and region segmentation results, achieving a multi-task learning / prediction effect.

[0104] Step 502: Determine the three-dimensional planar information of the multimedia information based on the region segmentation results and depth estimation results.

[0105] When determining the three-dimensional planar information, three-dimensional planar fitting can be performed based on the region segmentation results and depth estimation results to obtain the three-dimensional planar information of multimedia information.

[0106] This invention proposes that, after step 502, the method may further include: adjusting the determined three-dimensional plane information based on the semantic information and spatial relationship information corresponding to the determined three-dimensional plane information.

[0107] Specifically, based on the semantic and spatial relationship information corresponding to the determined three-dimensional plane information, the correlation between the three-dimensional planes is determined. Through the determined correlation, the determined three-dimensional plane information is adjusted to correct erroneous three-dimensional plane information.

[0108] The aforementioned relationships utilize spatial and semantic information from a three-dimensional plane. Unlike existing technologies that only utilize two-dimensional information, this embodiment extends spatial relationships to three dimensions, using the depth estimation results obtained in step 501. Compared to two-dimensional spatial relationships, three-dimensional spatial relationships more accurately reflect the precise positions between planes, such as... Figure 15 As shown in (a), this is a schematic diagram of a two-dimensional spatial relationship, in which plane A is located above plane B; as Figure 15 (b) is a schematic diagram of the three-dimensional spatial relationship. In the actual three-dimensional spatial relationship, plane A is perpendicular to plane B.

[0109] The aforementioned relationships between three-dimensional planes can be achieved using conditional random fields (CRFs). Specifically, each three-dimensional plane is treated as a fixed point, and all fixed points are connected to form a directed graph. Based on the CRF framework, conditional relationships are defined as three-dimensional spatial relationships and semantic connections, thus correcting the planar and semantic information of each plane. For example... Figure 16 As shown, before correction, a wall on the left was incorrectly classified as a floor. Through conditional random field, it was detected that the plane was surrounded by walls, so it was inferred that the plane information was likely incorrect. Through correction, the semantic information of the plane was correctly corrected to be a wall.

[0110] The relationships between the aforementioned three-dimensional planes can also be achieved in other ways, such as through Markov fields.

[0111] This invention proposes a hybrid method for determining planes. When determining planes in multimedia information, it adaptively selects an appropriate plane determination method based on the texture information of each region. Existing Simultaneous Localization and Mapping (SLAM) plane determination methods perform well in areas with rich texture, but they cannot obtain accurate results for areas lacking texture information (too little or no texture information), such as smooth desktops, glass surfaces, and walls, which are unsuitable for SLAM plane determination due to insufficient or no texture information. The deep learning-based plane determination method mentioned in this invention, for areas lacking texture information, can further determine the plane's normal vector, orientation, and other information based on the obtained 3D plane information. This information is highly beneficial for subsequent rendering of virtual display information, making the generated AR display information more realistic and improving the user experience.

[0112] Specifically, before performing region segmentation and depth estimation on multimedia information, the process may further include: determining the texture information of the multimedia information; and determining the texture-deficient regions based on the texture information.

[0113] Subsequently, for the identified texture-deficient regions, region segmentation and depth estimation are performed according to the method proposed in the embodiments of this invention.

[0114] In determining the texture information of multimedia information, the number of feature points in a region of the multimedia information (which can be determined based on user operation) can be determined first to judge whether the texture information of the region is rich. A threshold T is set. When the number of feature points exceeds the threshold T, the region is determined to be a texture-rich region; otherwise, it is a texture-deficient region.

[0115] like Figure 17 The diagram illustrates the process of determining the plane using a hybrid method. For the input image, the richness of texture information is determined. If the texture information is rich, the SLAM plane determination method is selected to determine the plane in the image. If the texture information is too little or nonexistent, the deep learning network-based plane determination method in this embodiment of the invention is selected for region segmentation and depth estimation. Based on the region segmentation and depth estimation results, three-dimensional plane fitting is performed to finally obtain the three-dimensional plane information.

[0116] The method of determining the hybrid plane described above can be performed on multimedia information, or it can be performed on certain areas. For example, the method of determining the hybrid plane described above can be performed on the area where the user wants to place virtual display information (the area where virtual display information needs to be rendered).

[0117] The processing results of the above-mentioned method for determining the plane by mixing are as follows: Figure 18 As shown in the image, the bright spots represent texture feature points; the denser the texture feature points, the richer the texture information in that area. The two squares in the image represent virtual display information. The user chooses to place the squares on a desktop without texture information, according to... Figure 17 The steps involved determine that the area lacks texture information, necessitating the use of a deep learning-based planar determination method instead of the existing traditional SLAM planar determination method. This deep learning-based method yields information such as the desktop's normal vector, orientation, and size. Combining this information with virtual display information allows for more realistic rendered AR displays, enhancing the user experience.

[0118] The aforementioned hybrid plane determination method enhances its applicability across different scenarios. It applies the SLAM plane determination method to texture-rich regions and the deep learning-based plane determination method (region segmentation, depth estimation, and plane fitting) to texture-deficient regions, thus compensating for the shortcomings of the SLAM method. This hybrid plane determination method enables plane estimation in any scene, significantly increasing its applicability while overcoming the deficiencies of traditional methods and leveraging their advantages.

[0119] This invention determines the three-dimensional plane of multimedia information by performing region segmentation and depth estimation. Compared with the existing two-dimensional plane determination method, the determined three-dimensional plane is more accurate. Subsequently, augmented reality display information is displayed based on the determined three-dimensional plane information, which can improve the realism of the augmented reality display results and thus enhance the user experience.

[0120] Example 2

[0121] This invention proposes a method for displaying augmented reality information, such as... Figure 19 As shown, the method includes the following steps:

[0122] Step 1001: Determine the three-dimensional planar information corresponding to the multimedia information; Step 1002: Display augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information.

[0123] In this embodiment of the invention, it is proposed that the three-dimensional planar information corresponding to multimedia information can be determined by, but is not limited to, the planar determination method described in Embodiment 1 above.

[0124] Further, step 1002 includes step 10021 (not labeled in the figure) and step 10022 (not labeled in the figure), wherein step 10021 is to obtain the attribute information corresponding to the three-dimensional plane information and / or the attribute information corresponding to the virtual display information; step 10022 is to display the augmented reality display information according to the obtained attribute information corresponding to the three-dimensional plane information and / or the attribute information corresponding to the virtual display information.

[0125] The attribute information corresponding to the three-dimensional plane includes at least one of the following: semantic information corresponding to the three-dimensional plane, associated attribute information corresponding to the semantic information, and physical attribute information of the three-dimensional plane.

[0126] In this embodiment of the invention, attribute information corresponding to the three-dimensional planar information and attribute information corresponding to the virtual display information can be acquired simultaneously to generate augmented reality display information. Alternatively, only the attribute information of the virtual display information can be acquired to generate augmented reality display information. For example, if the attribute information of the virtual display information determines that the virtual display information is a flying animal, then the virtual display information can be displayed at any location. That is, in this case, only the augmented reality display information corresponding to the virtual display information can be determined, while the three-dimensional planar information corresponding to the multimedia information can be left undetermined.

[0127] Among these, the aforementioned related attribute information is related to semantic information and is derived from it; semantic information can contain related attribute information. For example, if the semantic information is "sea surface," then "swimmable" is a related attribute of "sea surface."

[0128] Furthermore, physical attribute information may include at least one of the following: area, color, contrast, texture, etc.

[0129] Furthermore, the steps for obtaining semantic information corresponding to the three-dimensional planar information include any one of steps A (not labeled in the figure), B (not labeled in the figure), and C (not labeled in the figure):

[0130] Step A: Use the semantic information corresponding to the two-dimensional planar information of the multimedia information as the semantic information of the corresponding three-dimensional planar information; Step B: Determine the semantic information of the three-dimensional planar information based on the semantic information corresponding to the two-dimensional planar information of the multimedia information and the depth estimation results of the multimedia information; Step C: Perform semantic analysis on the three-dimensional planar information to obtain the semantic information corresponding to the three-dimensional planar information.

[0131] This invention also proposes that the contextual semantic information of the three-dimensional plane information can be determined, and then the semantic information corresponding to the three-dimensional plane can be adjusted based on the aforementioned contextual semantic information, thereby improving the accuracy of the determined semantic information of the three-dimensional plane.

[0132] When displaying AR information, it is necessary to determine how the virtual display information should be displayed. This embodiment of the invention proposes that the display position of the virtual display information can be automatically recommended. Step 10022 includes: determining the positional relationship between the virtual display information and the three-dimensional plane, and / or the positional relationship between the virtual display information, based on the attribute information corresponding to the acquired three-dimensional plane information and / or the attribute information corresponding to the virtual display information; and displaying the augmented reality information according to the determined positional relationship.

[0133] The diagram illustrating the display location of automatically recommended virtual display information is shown below. Figure 20 As shown, Figure 20 In (a), the three-dimensional planar information obtained from the multimedia information includes the desktop and the ground, and the virtual display information is a teacup. According to the attribute information of the two, the teacup can be placed on the desktop or on the ground. It is generally not placed on the side wall of the desktop, but it is more common to place it on the desktop. Therefore, based on the above relationship, it is recommended to place the teacup on the desktop. Figure 20 In (b), when there are multiple virtual display information, the relative positional relationship between the multiple virtual display information can also be automatically recommended. For example, if both the monitor and the keyboard are virtual display information and both are placed on the desktop, it is less common to place the keyboard behind the monitor and more common to place the keyboard in front of the monitor. Therefore, it is automatically recommended to place the keyboard in front of the monitor.

[0134] The aforementioned automatic recommendation method can be implemented based on knowledge-based filtering systems and long short-term memory neural networks (LSTM). For example... Figure 21 As shown, firstly, multimedia information and virtual display information are determined. The multimedia information is divided into several regions in a grid pattern. The virtual display information is placed in a certain position in the grid. Several combinations of display information are randomly constructed. Some combinations that do not conform to the rules are eliminated by a prior knowledge filter. Finally, the remaining combinations are scored by vertical LSTM and horizontal LSTM. The combination with the highest score is the automatically recommended placement position.

[0135] Reference Figure 22As shown, the method mentioned in this embodiment of the invention is used to perform region segmentation and depth estimation on multimedia information. Based on the region segmentation and depth estimation results, three-dimensional plane fitting is performed to obtain three-dimensional plane information. According to the attribute information corresponding to the three-dimensional plane and / or the attribute information corresponding to the virtual display information, it can be determined that the virtual display information (such as land animals) cannot fly in the air or walk on walls; it can only walk on the ground. Therefore, a plane belonging to the ground can be selected as the fusion object. Finally, the virtual display information is placed face up on the ground, instead of being directly suspended in the air as in existing technologies. Furthermore, the display volume and specific position of the virtual display information in the multimedia information can be determined based on the actual size of the virtual display information and the distance of the image, to avoid discrepancies between the virtual display information and the real situation in the generated augmented reality display information.

[0136] In the augmented reality display information display method proposed in this embodiment of the invention, the display mode (including position, size, etc.) of the virtual display information is comprehensively determined based on the attribute information of the virtual display information and / or the attribute information of the three-dimensional plane information. Therefore, the obtained augmented reality display information can better match the real situation and improve the user experience.

[0137] The following example, using furniture placement in a user's daily life, illustrates the solution for automatically recommending the display location of virtual information.

[0138] In this example, the virtual display information may specifically include furniture.

[0139] The above-described solutions of this invention can provide furniture layout preview and automatic recommendation of furniture display positions, including previewing the furniture placement effect, inferring reasonable positions where specific furniture can be placed, and the optimal placement position for furniture.

[0140] Furniture arrangement is a basic living need for users. When purchasing furniture or changing the overall layout, it's necessary to plan how to arrange the furniture and place it in designated locations beforehand. However, users usually only know whether the furniture layout is reasonable or aesthetically pleasing after it's been arranged, making it difficult to achieve a satisfactory layout and significantly increasing the cost of changing the layout. Each time the layout is changed, users need to move the furniture to a designated location, which greatly increases their physical and mental exertion. In addition, similar to trying on clothes, some users are not satisfied with the actual arrangement of the furniture after purchasing it, or they don't have a suitable place to put it.

[0141] This example provides a furniture layout preview, allowing users to preview the layout effect before making changes. Users can also arrange furniture in advance when purchasing it to determine if it is suitable and if there is a suitable place in their home. Users can also follow the recommendations in this example to place furniture in reasonable or optimal locations.

[0142] The specific process for automatically recommending the display location of virtual display information is as follows: Figure 23 As shown, where,

[0143] Step 11: Perform region segmentation and depth estimation on the input multimedia information (image). Based on the region segmentation results, determine the three-dimensional plane information corresponding to the multimedia information. Then, based on the semantic information of the three-dimensional plane, filter out the three-dimensional planes related to furniture placement, such as wall planes, floor planes, etc.

[0144] The three-dimensional planar information corresponding to multimedia information includes semantic information (walls, floors, etc.) and physical information (size, shape).

[0145] In this example, it is proposed that the three-dimensional planar information corresponding to multimedia information can be determined by, but is not limited to, the planar determination method described in Embodiment 1 above.

[0146] Step 12: Obtain the furniture information to be placed (including image information, size information, etc.), match the 3D planar information with the furniture information, and set certain filtering rules, including semantic matching rules (e.g., the table can be placed on the floor but not on the wall) and size matching rules (e.g., the planar size must be larger than the furniture size). The matched area is the reasonable area for placing the furniture.

[0147] There are two ways to obtain information about the furniture to be placed. The first way is to obtain multimedia information containing actual furniture information, determine the three-dimensional plane of the multimedia information, extract the plane corresponding to the furniture separately, and determine the relevant information (images and dimensions, etc.). The second way is provided by the furniture seller, who directly provides the electronic information of the furniture, including images and dimensions.

[0148] A suitable area can be displayed on the screen (the display method can be to highlight the area with different colors) so that users can select the plane on which to place furniture. After the plane is determined, the plane and furniture information can be merged together to achieve a preview effect.

[0149] Step 13: If the user selects the option to automatically recommend the display location of virtual information, then within the above-mentioned reasonable area, the best placement location can be automatically recommended based on prior knowledge.

[0150] Once the placement plane is determined, the plane and furniture information can be integrated to create a preview. The furniture information is displayed virtually in the recommended locations based on their spatial relationships.

[0151] Reference Figure 24 This diagram illustrates the automatically recommended display location for virtual information. To preview how a TV table (virtual display information) would look in the living room, a user takes a picture of the living room using their mobile phone. The TV table information can be provided by the furniture supplier. Step 11 above uses geometric estimation (region segmentation and depth estimation, determining the corresponding 3D planar information based on the region segmentation results) to identify multiple planes in the room, including the floor, windows, TV screen, and walls. Steps 11 and 12 then find suitable placement locations within these planes. For example, if the floor is a suitable plane for a table and large enough to accommodate it, it's considered a suitable placement. Step 13, combined with prior knowledge, determines that TV tables are typically placed next to the TV, recommending the optimal placement as close to the TV as the floor. The optimal placement location is then integrated with the furniture information to achieve the preview effect.

[0152] If the user chooses to place the item according to the recommended location, the recommended location will be selected as the placement location. Users can also choose a custom placement option, manually selecting a specific plane as the final placement surface.

[0153] Example 3

[0154] In this embodiment of the invention, step 1002 further includes: determining a target plane in the three-dimensional plane information corresponding to the multimedia information; determining adjustment information corresponding to the target plane; and displaying augmented reality display information corresponding to the adjustment information.

[0155] The adjustment information may include, but is not limited to, adjustment direction information and / or adjustment angle information.

[0156] The first example of Example 3

[0157] This example provides a planar position adjustment recommendation method, including positional relationship awareness, and provides adjustment information to make the planar position meet the requirements.

[0158] Applicable Scenarios: When arranging furniture, clocks, picture frames, and other items, users need to ensure that the items are horizontal or vertical. For example, the coffee table and sofa in the living room are parallel, and the overall layout of the room follows a relatively strict vertical parallel relationship, with clocks and picture frames placed horizontally on the wall. However, because humans have a poor ability to perceive vertical and / or horizontal relationships and cannot accurately distinguish small angular differences, especially at close range, when arranging items such as picture frames, users often observe them from a distance after placement and then return to adjust them, or ask others if they are horizontal or vertical. This back-and-forth adjustment consumes extra energy and time for users and may require additional assistance. The planar position adjustment method provided in this example can help users accurately determine the angular relationship between planes, compensating for the human inability to accurately distinguish small angular differences. By accurately distinguishing small angular differences, the horizontal and / or vertical relationships between planes can be precisely determined, providing convenience in furniture placement, item placement, and interior layout, reducing the time and energy spent on multiple adjustments and improving the perfection of the layout.

[0159] This example proposes that, before determining the adjustment information corresponding to the target plane, the following steps are also included: determining the reference plane and the positional relationship between the target plane and the reference plane in the three-dimensional plane information corresponding to the multimedia information;

[0160] Determine the adjustment information corresponding to the target plane, including: determining the position adjustment information of the target plane based on the determined positional relationship, as the adjustment information corresponding to the target plane.

[0161] The adjustment information refers to suggestions for adjusting the position of the target plane. The target plane is the plane whose position needs to be adjusted. The positional relationship between the target plane and the reference plane can include the current positional relationship or the target positional relationship. The positional relationship between the target plane and the reference plane can be, but is not limited to, the angle between the edge lines of the planes.

[0162] In this example, the target plane, and / or reference plane, and / or target positional relationship can be selected and determined by the user. For instance, when a user hangs a picture frame on a wall and wants it to be parallel to the ground or ceiling, the user can choose the picture frame as the target plane, the ceiling as the reference plane, and the target positional relationship as the frame's edge parallel to the ceiling edge. Alternatively, the user can determine the target plane, and then the reference plane or target positional relationship can be automatically determined based on it. For example, if the target plane is the picture frame, and most users want the picture frame to be parallel to the ground or ceiling when hanging it, the ground or ceiling plane can be automatically used as the reference plane, and the target positional relationship can be set to the frame's edge parallel to the ceiling edge.

[0163] like Figure 25The diagram shown is a flowchart illustrating the planar position adjustment recommendation method in this example.

[0164] Step 21: Perform region segmentation and depth estimation on the input multimedia information (image). Based on the region segmentation results, determine the corresponding 3D planar information of the multimedia information. Step 22: Determine the current positional relationship between the target plane and the reference plane.

[0165] Users can manually select a target plane and a reference plane in a 3D plane, such as the canvas frame and the ceiling plane, which are the target plane and the reference plane, respectively.

[0166] Based on the 3D planar information, the top border of the oil painting frame plane and the edge line of the ceiling are obtained. The angle between the top border of the oil painting frame plane and the edge line of the ceiling is calculated. This angle is used as the current positional relationship between the target plane and the reference plane, or as the planar angle between the target plane and the reference plane. Therefore, the precise positional relationship between the target plane and the reference plane can be obtained.

[0167] Step 23: Based on the precise current positional relationship between the target plane and the reference plane, as well as the target positional relationship, determine the adjustment information (target plane position adjustment suggestion) to assist the user in completing the target plane position adjustment.

[0168] In step 22, the user has already selected the target plane and the reference plane. The user can further select the desired positional relationship between the target plane and the reference plane, such as the frame of the picture being parallel to the edge of the ceiling. Based on the current positional relationship obtained in step 22, the angle between the edge lines of the two planes is 3°, while the selected target positional relationship is 0°. Therefore, the system automatically provides an adjustment suggestion to rotate the target plane by 3°. The user adjusts the position of the target plane according to the adjustment suggestion, which makes the edge lines of the two planes parallel, ultimately allowing the picture frame to be hung horizontally on the wall.

[0169] The recommended planar position adjustment in this example is shown below. Figure 26 and Figure 27 As shown.

[0170] Reference Figure 26 In daily life, users may need to place items such as picture frames horizontally on the wall. However, people have poor subjective perception of vertical and horizontal relationships, especially for small angles (such as 5°). Generally, users need to hang the picture frame on the wall first, then observe from a distance whether it is horizontal, and then make adjustments. This may require multiple adjustments, and it is not guaranteed that the picture frame will be placed horizontally in the end.

[0171] In this scenario, users collect multimedia information (images or videos) through multimedia acquisition devices (such as cameras) on their terminals (mobile phones, AR glasses, etc.).

[0172] Step 21 allows for the determination of planes within the acquired multimedia information using the aforementioned method. This enables users to select target and reference planes. Users can select the segmented plane as both the target and reference planes by clicking on the location of the plane on the screen. Users can specify the frame as the target plane and the ceiling as the reference plane via touchscreen on their mobile phone or AR glasses display.

[0173] After selecting two planes, the display screen offers two options: horizontal and vertical. Users can choose to keep the two planes horizontal or vertical, or to keep their edges horizontal or vertical. When the user selects the horizontal edge option, it means the target plane and the reference plane are parallel, i.e., the angle between the top edge of the painting frame plane and the edge of the ceiling is 0°. Step 22 shows that the current positional relationship between the two planes is: the angle between the top edge of the painting frame plane and the edge of the ceiling is 5°.

[0174] Since the target's included angle is 0°, while the current included angle is 5°, they are in a non-parallel state. Therefore, the display shows the user "Angle = 5°" between the edges of the selected plane.

[0175] After obtaining the adjustment suggestion in step 23, rotate the target plane counterclockwise by 5° on the wall. After adjusting the frame according to the adjustment suggestion, the angle between the top edge of the oil painting frame and the edge of the ceiling can be 0°. The adjustment suggestion can be prompted to the user through text prompts on the display screen or voice prompts, instructing the user to "rotate counterclockwise by 5°" according to the current adjustment suggestion.

[0176] In this example, once the user selects the target plane, the reference plane, and the target positional relationship between the planes, the positional relationship between the target plane and the reference plane can be obtained periodically, and adjustment suggestions can be provided.

[0177] After the user rotates according to the adjustment suggestions, they can use their mobile phone or AR glasses to continuously take pictures and determine the changes in positional relationship in real time. For example, if the user rotates too much, such as 7° counterclockwise, the system will reassess the current positional relationship and remind the user to rotate 2° clockwise until the target positional relationship is reached, at which point the reminder will stop.

[0178] In this example, you can also choose to evaluate the current positional relationship in real time. In this case, while making adjustments, the mobile phone or AR glasses need to continuously capture the current scene. The user can slowly rotate the oil painting and display the current positional relationship in real time until the target positional relationship is reached. Then, the adjustment is completed through the display screen or voice prompts.

[0179] Reference Figure 27 Users need to ensure the sofa is perpendicular to the wall. However, since the sofa is not directly against the wall, maintaining this perpendicularity is difficult. Figure 26 Similarly, a mobile phone or AR glasses captures the current scene to obtain multimedia information. The user selects the side of the sofa and the wall as the target plane and reference plane, respectively, and selects the target spatial relationship as follows: the bottom edge of the sofa side is perpendicular to the edge of the wall, that is, the angle between the bottom edge of the sofa side and the edge of the wall is 90°. Through step 2, the current positional relationship between the sofa side and the wall can be obtained, that is, the current angle between the bottom edge of the sofa side and the edge of the wall. The system determines the angle that needs to be adjusted, and the display screen or voice prompts the user to adjust according to the adjustment suggestions until the target positional relationship is achieved.

[0180] Alternatively, a simpler method can be chosen where only the wall is selected as the reference plane, without selecting the target plane. The normal vector of the wall is first obtained, and the normal vector line is displayed on the screen. The user continuously observes the screen and places the sofa along the normal vector line.

[0181] The second example of Example 3

[0182] This example provides a recommended method for adjusting the posture of the acquisition device, which can prompt users to adjust the posture of the multimedia acquisition device (camera, mobile phone, etc.) to obtain the optimal acquisition angle.

[0183] Applicable scenarios: When users take photos with a camera or mobile phone, they usually hope to get a picture of a certain object that is facing them directly. For example, when taking pictures of documents or oil paintings, if the photos are rotated or tilted, it will cause inconvenience for subsequent reading. However, it is not intuitive for users to know how to adjust the camera posture in order to get a picture that is facing them directly.

[0184] This example can analyze the collected multimedia information and present the posture adjustment suggestions of the acquisition device on the display screen. By rotating or moving the acquisition device according to the adjustment suggestions, the user can obtain the multimedia information in the correct orientation.

[0185] This example proposes that, before determining the adjustment information corresponding to the target plane, the following steps are also included: determining the positional relationship between the target plane and the acquisition plane corresponding to the acquisition device for acquiring multimedia information;

[0186] Determine the adjustment information corresponding to the target plane, including: determining the attitude adjustment information of the acquisition plane based on the determined positional relationship, as the adjustment information corresponding to the target plane.

[0187] Among them, suggestions for adjusting the attitude of the information acquisition equipment are provided. The target plane is the plane corresponding to the object to be photographed.

[0188] In the examples below, the target plane can specifically refer to the plane corresponding to the object to be photographed, such as a document or an oil painting.

[0189] In the following embodiments, the acquisition device for collecting multimedia information may be a camera or a mobile phone, and the acquisition plane corresponding to the acquisition device for collecting multimedia information may be the plane corresponding to the camera or mobile phone.

[0190] When the adjustment information is angle information, the angle information may specifically include, in the following examples, adjusting the rotation angle and / or tilt angle.

[0191] like Figure 28 The diagram shown is a schematic of the recommended method for adjusting the attitude of the data acquisition device in this example.

[0192] Step 31: Perform region segmentation and depth estimation on the input multimedia information (such as images), and determine the three-dimensional planar information corresponding to the multimedia information based on the region segmentation results.

[0193] Users can manually select a target plane in a three-dimensional plane, for example, when photographing an oil painting, the plane of the oil painting can be used as the target plane.

[0194] Step 32: Determine the relative normal vector information of the target plane with respect to the acquisition plane.

[0195] If the target plane and the acquisition plane are parallel, that is, the target plane is acquired in the frontal direction, then the relative normal vector in three-dimensional space has a fixed standard value (such as 1,0,0). If they are not parallel, it can be assumed that the target plane is being acquired while rotating, and the relative normal vector will have other values.

[0196] The positional relationship between the target plane and the acquisition plane can be determined using this relative normal vector information.

[0197] Step 33: Based on the positional relationship between the target plane and the acquisition plane, the attitude adjustment information of the acquisition plane can be determined, i.e., attitude adjustment suggestions, which can specifically be adjusting the direction and / or adjusting the angle.

[0198] At this point, the adjustment angle can be the rotation angle. A rotation indicator, including the rotation direction and angle, can be displayed on the screen. After adjusting the acquisition device according to the rotation indicator, the user can obtain the target plane viewed directly.

[0199] Furthermore, the positional relationship between the target plane and the acquisition plane can be determined based on their edge lines. If the angle between the two planes is not 0°, it can be assumed that the target plane is being acquired at an angle. Based on the positional relationship between the target plane and the acquisition plane, the attitude adjustment information of the acquisition plane, i.e., attitude adjustment suggestions, can be determined, specifically adjustment direction and / or adjustment angle. The adjustment angle in this case can be the tilt angle. A tilt indicator, including the tilt direction and tilt angle, can be displayed on the screen. After the user adjusts the acquisition device according to the tilt indicator, the target plane can be viewed directly.

[0200] The recommended method for adjusting the attitude of the data acquisition device in this example is illustrated in the diagram below. Figure 29 As shown. The user opens the phone's camera, and the image captured is as follows. Figure 29 As shown in (a), there is an oil painting in the image, but the painting has a certain rotation angle. If the user is not satisfied with the shooting result, they can choose the front-view mode. In this mode, the plane information is obtained through step 31 to obtain the oil painting plane; through step 32, the relative normal vector of the oil painting plane is obtained. Since the relative normal vector of the oil painting plane is not a fixed standard value, it can be automatically determined that the image obtained by taking the picture has a rotation angle. According to the rotation angle, combined with the camera posture, the camera posture adjustment suggestion is to rotate the camera in the opposite direction of the rotation angle. The adjustment suggestion can be displayed on the screen, such as... Figure 29 As shown in (b), the user can obtain a front-view image by rotating the camera according to the adjustment suggestions. When the user adjusts the camera according to the suggestions, the system can update the current angle in real time. When the relative normal vector reaches a fixed standard value, the system can prompt the user that the adjustment is complete. Figure 29 As shown in (c).

[0201] Similarly, such as Figure 29 (d), 29(e), 29(f): When the angle between the edge line of the target plane (document) and the acquisition plane is not 0°, it is not convenient for the user to view. In this case, it is recommended that the user tilt the phone according to the adjustment suggestions so that the content can be captured from the front.

[0202] In this example, tilting the acquisition device means moving the acquisition device within the plane containing the acquisition plane, while rotating the acquisition device means adjusting the roll angle, pitch angle, and yaw angle with the center of the acquisition device as the origin.

[0203] Example 4

[0204] In this embodiment of the invention, step 1002 further includes: determining a driving avoidance plane in the three-dimensional planar information corresponding to the multimedia information; determining driving assistance information based on the driving avoidance plane; and displaying augmented reality display information corresponding to the driving assistance information.

[0205] The first example of Example 4

[0206] This example provides a method for determining whether a vehicle can pass through a narrow area in a driver assistance system.

[0207] One common problem to solve when driving is navigating narrow lanes or alleys. Users need to estimate the accurate width of the passable lane to determine if their vehicle can pass smoothly. However, human estimations of width are often rough, and due to inherent human visual limitations, they may even misjudge the width of their vehicle and the lane. This example utilizes machine vision to provide more accurate visual estimation results, offering users supplementary information (such as vehicle width, lane width, and whether it is passable) to help them determine whether they can pass through narrow areas ahead. Furthermore, this solution can be applied to driver assistance systems or autonomous driving systems, addressing one type of problem in automated driving.

[0208] If the driving avoidance plane is the obstacle plane on both sides of the driving road, then driving assistance information is determined based on the driving avoidance plane, including: determining the width information of the driving road based on the obstacle plane on both sides of the driving road; and determining whether the prompt indicates whether the driving road can be traversed based on the width information of the driving road, as driving assistance information.

[0209] Specifically, the obstacle planes on both sides of the road surface can be the planes on both sides of the alley or the obstacle planes on both sides of the narrow lane as described in the following embodiments, or they can be walls or other planes corresponding to vehicles.

[0210] The specific process is as follows: Figure 30 As shown, where,

[0211] Step 41: Determine the three-dimensional planar information corresponding to the multimedia information.

[0212] In this example, it is proposed that the three-dimensional planar information corresponding to multimedia information can be determined by, but is not limited to, the planar determination method described in Embodiment 1 above.

[0213] Step 42: Based on the semantic information of the obtained three-dimensional plane, determine the obstacle planes on both sides of the road surface, such as the walls on both sides of the road surface, the side planes of other vehicles, etc., and calculate the width information of the road surface, which can also be called the road width or the actual road surface width.

[0214] When determining the plane of an obstacle, it is necessary to determine the plane closest to the current vehicle. For example, if there are other vehicles on both sides of the road, the plane where the rearview mirrors of the other vehicles are located can be used as the obstacle plane.

[0215] Step 43: Based on the actual road width and the vehicle's own attributes (such as vehicle width), determine whether the vehicle can pass. If the actual road width is greater than the vehicle's own width, the user can be prompted that the vehicle can pass.

[0216] Reference Figure 31 The driver assistance system uses a front camera to capture road information and obtain a road condition image. Step 41 detects various planes in the road image, step 42 calculates the actual road width, and step 43 compares it with the vehicle width to obtain driver assistance prompts indicating whether the road is passable. These prompts can be used as recommended content and presented to the user via voice broadcast or displayed on the in-vehicle screen.

[0217] The second example of Example 4

[0218] This example provides a road condition estimation method in a driver assistance system, which can automatically determine the road conditions ahead and provide driver assistance prompts while the vehicle is in motion.

[0219] In autonomous or assisted driving systems, assessing road conditions ahead is a fundamental requirement, enabling timely adjustments to driving speed and other actions. A lack of this basic ability to judge road conditions can lead to catastrophic driving accidents.

[0220] For example, if the road ahead is in poor condition with many potholes, it's necessary to slow down. Driving at full speed could easily lead to a driving accident. This example only requires a common, low-cost optical camera to estimate road conditions and determine whether the road ahead is smooth and whether slowing down is necessary.

[0221] If the driving avoidance plane is the plane to be avoided on the driving road surface, then driving assistance information is determined based on the driving avoidance plane, including: driving suggestion information is determined based on the plane to be avoided on the driving road surface, as driving assistance information.

[0222] In the following embodiments, the plane to be avoided can be a plane that is not suitable for passage, such as a water surface, a pit surface, or an obstacle surface.

[0223] The specific process is as follows: Figure 32 As shown, where,

[0224] Step 51: Determine the three-dimensional planar information corresponding to the multimedia information.

[0225] In this example, it is proposed that the three-dimensional planar information corresponding to multimedia information can be determined by, but is not limited to, the planar determination method described in Embodiment 1 above.

[0226] Step 52: Based on the obtained three-dimensional planar information, extract the road surface and other unsuitable planes for passage (such as water surfaces, pit surfaces, and obstacle planes), and classify the planes into dangerous categories based on their depth information. Shallow pits are classified as low-risk, while deep pits, large amounts of water, and tall obstacles are classified as high-risk.

[0227] Step 53: Assess the danger situation ahead based on the danger level and provide driving advice information, i.e., driving advice. This driving advice can be a hazard warning or a replanned driving route.

[0228] The hazard level can be determined by comprehensively scoring the relevant planes based on their hazard classification level and area. The scoring rules can be weighted according to hazard level and plane size, with several score thresholds set. Exceeding a certain threshold will trigger different hazard warnings. For example, if there are potholes ahead with a low hazard level and a small area, a low-level hazard warning will be issued after comprehensive scoring, advising the driver to slow down. If there are numerous obstacles ahead with a high hazard level and a large area, a high-hazard warning will be issued, advising the driver to stop and observe.

[0229] In addition, you can replan your driving route based on the road conditions ahead.

[0230] In step 53, the hazard warnings include slowing down and detouring. When the driving suggestion is to detour, the route needs to be replanned to avoid the obstacle area ahead. This example can provide path planning with information on the plane to be avoided and road surface information.

[0231] Reference Figure 33 The autonomous driving system uses a front-facing camera to capture road information, obtaining two images, one above the other on the left. In the upper left image, after steps 51 and 52, the relevant planar surfaces, including water and road surfaces, are obtained. After hazard classification, small areas of water are categorized as medium-risk obstacles, requiring avoidance or detour. The system can then replan the driving route and display hazard warnings and the replanned route to the user. Figure 33 As shown in the attached diagram at the top right; similarly, in Figure 33 The image on the lower left shows information such as the road surface and shallow potholes. In the hazard classification, shallow potholes are classified as low-hazard. Since no avoidance route can be planned for the road conditions ahead, the user can be advised to slow down.

[0232] Example 5

[0233] This example provides a method for implementing an AR keyboard. Using AR devices, an English keyboard can be transformed into a multilingual keyboard, and a standard PIN pad can be converted into an encrypted one. Since standard keyboards are typically in English, users of different languages ​​often require keyboards in other specific languages, such as Russian or Korean. Because these other language keyboards are relatively harder to obtain than English keyboards, this causes significant inconvenience. Even if a user has multiple keyboards in different languages, constantly switching between them when needing to input multiple languages ​​is extremely cumbersome. Furthermore, random PIN pads are often more secure than fixed PIN pads when entering passwords. Random PIN pads are widely used in online transactions, but in offline transactions (card swiping, cash withdrawals), physical PIN pads are still required. If someone else sees a user's PIN while they are entering it, account security will be compromised.

[0234] The AR keyboard provided in this example can be used as a multilingual keyboard. Depending on the AR presentation method, it can present a multilingual switchable keyboard. The AR keyboard can also be used as a random password keyboard. According to the encoding rules, it presents an encrypted password keyboard through AR. Only the user can see it. Even if others see it, they cannot obtain the real password.

[0235] In this embodiment of the invention, step 1002 further includes: determining the planes where virtual display information needs to be displayed from the three-dimensional planar information corresponding to the multimedia information; obtaining the virtual display information corresponding to each plane where virtual display information needs to be displayed; and displaying the augmented reality display information corresponding to the virtual display information.

[0236] Among them, the plane that needs to display virtual display information is the plane whose semantic information is the key; the virtual display information is the key value information.

[0237] The method further includes: detecting user operations on a three-dimensional plane in multimedia information; determining user operation instructions based on the actual and virtual display information of the three-dimensional plane corresponding to the operation; and executing the corresponding operation based on the user operation instructions.

[0238] In the examples below, the plane of the key can specifically refer to the plane on which the key is located, which can be a regular keyboard or a password keyboard.

[0239] The implementation process of the AR keyboard in this example is as follows: Figure 34 As shown.

[0240] Step 61: Determine the three-dimensional plane information corresponding to the multimedia information, and filter out the key plane based on the obtained three-dimensional plane information.

[0241] In this example, it is proposed that the three-dimensional planar information corresponding to multimedia information can be determined by, but is not limited to, the planar determination method described in Embodiment 1 above.

[0242] When filtering key planes, planes whose semantic information is key can be retained and other irrelevant planes can be removed according to the filtering rules.

[0243] You can also create a digital model of the keyboard based on its layout to obtain a 3D digital model of the actual keyboard.

[0244] Step 62: Replace the original keyboard key values ​​with the specified virtual display information. This involves first obtaining the virtual display information corresponding to each key plane that needs replacement, then rendering the virtual display information at the corresponding key plane location to obtain the AR display information. The virtual display information can be the replaced key value information.

[0245] When the original keys are language keys, a multilingual keyboard function can be achieved by replacing the key values ​​between different languages. When the original keys are password keys, a random password keyboard function can be achieved by replacing the key values.

[0246] By using three-dimensional planar information, the virtual display information can be rendered more effectively at the key position, resulting in a more realistic AR virtual keyboard.

[0247] Since the key value of the key has been replaced, when the user performs an operation on the key, the replaced virtual display information corresponding to that key, i.e. the replaced key value, can be determined based on the replaced key value. Based on the user's actual operation command, the corresponding operation can be executed.

[0248] For example, when an AR virtual keyboard is used to implement a random password keyboard, when a user presses a password on the keyboard, the bank system or the keyboard will decrypt the user's password based on the key values ​​before and after the replacement, obtain the user's real password, and thus complete the transaction.

[0249] The AR keyboard includes two types: a multilingual AR keyboard and an encrypted password keyboard. Figure 35 A schematic diagram of a multilingual AR keyboard is shown. Figure 35 In (a), the actual keyboard is an English keyboard, but through an AR device (such as a mobile phone or AR glasses), it is displayed as a Russian keyboard, such as... Figure 35 As shown in (b).

[0250] Figure 36 The diagram shows an encrypted keypad. When viewed through an AR device, the keypad is displayed as an encrypted keypad, which differs from the content of the real keypad seen by others.

[0251] This invention provides a method for displaying augmented reality information. Compared with the prior art, this invention determines the three-dimensional plane information corresponding to the multimedia information, and can obtain the attribute information corresponding to the three-dimensional plane information and / or the attribute information corresponding to the virtual display information. Augmented reality information is then displayed based on these two types of information. For example, the attribute information corresponding to the three-dimensional plane can be determined to be ground or water surface, and the attribute information corresponding to the virtual display information can be determined to be land animals or aquatic animals. Based on these attribute information, land animals can be added to the ground, and aquatic animals can be added to the water surface to display augmented reality information. This avoids the situation where the displayed virtual reality information shows land animals on the water surface or aquatic animals on the ground, thereby improving the realism of the augmented reality display results and enhancing the user experience.

[0252] This invention provides a planar determination device, such as... Figure 37 As shown, the device includes: a processing module 3701 and a first determining module 3702; wherein,

[0253] The processing module 3701 is used to perform region segmentation and depth estimation on multimedia information.

[0254] The first determining module 3702 is used to determine the three-dimensional planar information of multimedia information based on the region segmentation result and depth estimation result obtained by the processing module 3701.

[0255] This invention provides a plane determination device that can implement the method embodiments described above. For specific functional implementation, please refer to the description in the method embodiments, which will not be repeated here.

[0256] This invention provides a display device for displaying augmented reality information, such as... Figure 38 As shown, the device includes: a second determining module 3801 and a display module 3802, wherein,

[0257] The second determining module 3801 is used to determine the three-dimensional planar information corresponding to the multimedia information.

[0258] Display module 3802 is used to display augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information determined by the second determining module 3801.

[0259] This invention provides a display device for augmented reality information. Compared with the prior art, this invention determines the three-dimensional plane information corresponding to the multimedia information, and can obtain the attribute information corresponding to the three-dimensional plane information and / or the attribute information corresponding to the virtual display information. Augmented reality display information is generated based on the above two types of information. For example, the attribute information corresponding to the three-dimensional plane can be determined to be ground or water surface, and the attribute information corresponding to the virtual display information can be determined to be land animals or aquatic animals. Based on the attribute information of the two, land animals can be added to the ground and aquatic animals can be added to the water surface to generate augmented reality information. This avoids the situation where the generated virtual reality information is land animals on the water surface or aquatic animals on the ground, thereby improving the augmented reality display results and enhancing the user experience.

[0260] This invention provides a display device for augmented reality information, which can implement the method embodiments described above. For specific functional implementation, please refer to the description in the method embodiments, which will not be repeated here.

[0261] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0262] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0263] Those skilled in the art will understand that this invention includes devices for performing one or more of the operations described in this application. These devices may be specifically designed and manufactured for the desired purpose, or may include known devices found in general-purpose computers. These devices have computer programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a device (e.g., a computer)-readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. That is, a readable medium includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0264] Those skilled in the art will understand that each block in these structural diagrams and / or block diagrams and / or flow diagrams, as well as combinations of blocks in these structural diagrams and / or block diagrams and / or flow diagrams, can be implemented using computer program instructions. Those skilled in the art will also understand that these computer program instructions can be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing method for implementation, thereby enabling the processor of the computer or other programmable data processing method to execute the schemes specified in the blocks or plurality of blocks of the structural diagrams and / or block diagrams and / or flow diagrams disclosed herein.

[0265] Those skilled in the art will understand that the steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this invention can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0266] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for determining a plane, characterized in that, include: Using the same deep learning network, multimedia information is segmented into regions. The region segmentation results include two-dimensional plane information and the corresponding semantic information of the two-dimensional plane. Depth estimation is performed on the multimedia information; Based on the region segmentation results and depth estimation results, the three-dimensional planar information of the multimedia information is determined; Determine the texture information of multimedia information; Based on the texture information, determine the texture missing areas; Region segmentation and depth estimation of multimedia information include: For the identified texture-deficient regions, region segmentation and depth estimation are performed.

2. The method according to claim 1, characterized in that, The process of segmenting and estimating the depth of multimedia information also includes: Based on the region segmentation results, the depth estimation results are corrected; and / or The region segmentation results are corrected based on the depth estimation results.

3. The method according to claim 1, characterized in that, Also includes: The determined three-dimensional plane information is adjusted based on the semantic and spatial relationship information corresponding to the determined three-dimensional plane information.

4. The method according to claim 3, characterized in that, The adjustment of the determined three-dimensional planar information includes: Based on the semantic and spatial relationship information corresponding to the determined three-dimensional plane information, the association between the three-dimensional planes is determined; The determined three-dimensional planar information is adjusted based on the established correlation.

5. The method according to any one of claims 1-4, characterized in that, The deep learning network is trained by combining semantic annotation and planar annotation.

6. A method for displaying augmented reality information, characterized in that, include: The three-dimensional planar information corresponding to multimedia information is determined by the planar determination method according to any one of claims 1-5; Augmented reality display information is displayed based on the three-dimensional planar information corresponding to the multimedia information.

7. The method according to claim 6, characterized in that, The step of displaying augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information includes: Obtain the attribute information corresponding to the three-dimensional plane information, and / or the attribute information corresponding to the virtual display information; Augmented reality display information is displayed based on the attribute information corresponding to the acquired 3D planar information and / or the attribute information corresponding to the virtual display information.

8. The method according to claim 7, characterized in that, The attribute information corresponding to the three-dimensional plane includes at least one of the following: The semantic information corresponding to the three-dimensional plane, the associated attribute information corresponding to the semantic information, and the physical attribute information of the three-dimensional plane.

9. The method according to claim 8, characterized in that, Obtain the semantic information corresponding to the three-dimensional planar information, including: The semantic information corresponding to the two-dimensional planar information of the multimedia information is used as the semantic information corresponding to the three-dimensional planar information; Alternatively, the semantic information corresponding to the three-dimensional planar information can be determined based on the semantic information corresponding to the two-dimensional planar information of the multimedia information and the depth estimation of the multimedia information; Alternatively, semantic analysis can be performed on the three-dimensional planar information to obtain the semantic information corresponding to the three-dimensional planar information.

10. The method according to any one of claims 7-9, characterized in that, Based on the attribute information corresponding to the acquired 3D planar information and / or the attribute information corresponding to the virtual display information, augmented reality display information is displayed, including: Based on the attribute information corresponding to the acquired 3D planar information and / or the attribute information corresponding to the virtual display information, determine the positional relationship between the virtual display information and the 3D planar information, and / or the positional relationship between the virtual display information; Based on the determined positional relationships, augmented reality information is displayed.

11. The method according to claim 6, characterized in that, The step of displaying augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information includes: The target plane is determined from the three-dimensional planar information corresponding to the multimedia information; Determine the adjustment information corresponding to the target plane; Display the augmented reality information corresponding to the adjusted information.

12. The method according to claim 11, characterized in that, Also includes: In the three-dimensional planar information corresponding to the multimedia information, a reference plane is determined, as well as the positional relationship between the target plane and the reference plane; Determining the adjustment information corresponding to the target plane includes: Based on the determined positional relationship, the positional adjustment information of the target plane is determined, which serves as the adjustment information corresponding to the target plane.

13. The method according to claim 11, characterized in that, Also includes: Determine the positional relationship between the target plane and the acquisition plane corresponding to the acquisition device that acquires the multimedia information; Determining the adjustment information corresponding to the target plane includes: Based on the determined positional relationship, the attitude adjustment information of the acquisition plane is determined and used as the adjustment information corresponding to the target plane.

14. The method according to any one of claims 11-13, characterized in that, The adjustment information includes adjustment direction information and / or adjustment angle information.

15. The method according to claim 6, characterized in that, The step of displaying augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information includes: The driving avoidance plane is determined from the three-dimensional planar information corresponding to the multimedia information; Based on the driving avoidance plane, determine the driving assistance information; Display augmented reality information corresponding to the driving assistance information.

16. The method according to claim 15, characterized in that, The driving avoidance plane is the obstacle plane on both sides of the driving road; Based on the driving avoidance plane, driving assistance information is determined, including: Determine the width information of the driving road surface based on the plane of obstacles on both sides of the driving road surface; Based on the width information of the driving road surface, a prompt is determined to indicate whether the vehicle can pass through the driving road surface, and this prompt serves as the driving assistance information.

17. The method according to claim 15, characterized in that, The driving avoidance plane is the plane to be avoided on the driving road surface; Based on the driving avoidance plane, driving assistance information is determined, including: Based on the plane to be avoided on the road surface, driving suggestion information is determined as the driving assistance information.

18. The method according to claim 6, characterized in that, The step of displaying augmented reality display information based on the three-dimensional planar information corresponding to the multimedia information includes: In the three-dimensional planar information corresponding to the multimedia information, determine the plane on which the virtual display information needs to be displayed; Obtain the virtual display information corresponding to each plane that needs to display virtual display information; Display the augmented reality display information corresponding to the virtual display information.

19. The method according to claim 18, characterized in that, The plane that needs to display virtual display information is a plane whose semantic information is key; the virtual display information is key value information.

20. The method according to claim 18 or 19, characterized in that, Also includes: Detect operations targeting the three-dimensional plane within the multimedia information; The operation instruction is determined based on the actual and virtual display information of the three-dimensional plane corresponding to the operation; Perform the corresponding operation according to the operation instructions.

21. An electronic device, characterized in that, Including memory and processor; The memory contains computer programs; The processor is configured to execute the computer program to implement the method of any one of claims 1 to 20.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 20.

Citation Information

Patent Citations

  • Extra-large imaging depth three-dimensional display method based on optical 4f system

    CN103197429A

  • Method for building robot simulation drilling system based on reality scene

    CN104484522A

  • Methods and systems for augmented reality of 3D scenes based on depth information

    CN104539925B

  • Naked eye 3D augmented reality method and device based on transparent liquid crystals

    CN105763865A

  • Method and apparatus for representation 3D space based plane

    KR1020130118105A