Display method and device based on augmented reality

Abstract

The invention discloses a display method and device based on augmented reality, and the method comprises the steps: obtaining real scene image information; performing calculation processing on the real scene image information to obtain target scene information; and analyzing and processing the target scene information to obtain target display information. Visibly, the virtual search and rescue information is superposed in the real terrain sand table environment, so that the real world and the virtual information can be seen at the same time, and the cooperation efficiency and the response speed of the search and rescue emergency scheme can be improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality, and in particular to a display method and device based on augmented reality. Background Art

[0002] With the rapid development of information technology and digital media, the form of search and rescue scene display is also constantly innovating. Traditional exhibition displays mainly rely on static physical exhibits, graphic descriptions and video playback. Although this method is intuitive, it has certain limitations in information transmission and audience interaction. In recent years, virtual reality (VR) and augmented reality (AR) technologies have gradually been applied to the field of exhibitions to enhance the audience's sense of participation and experience.

[0003] Augmented reality (AR) technology provides visitors with a rich interactive experience by superimposing virtual information on real scenes. For example, visitors can view the three-dimensional model, animation demonstration or detailed information of the exhibits through dedicated AR glasses, which not only enriches the display content, but also makes the display effect more vivid and interesting.

[0004] Currently, there are two ways to display search and rescue emergency plans on the market: based on PC and VR (virtual reality). Their characteristics are as follows:

[0005] The PC-based display relies on static physical exhibits, graphic descriptions and video playback. Although this method is intuitive, the information presentation is relatively simple, lacks interactivity, and cannot provide an immersive experience. It is difficult for the audience to fully understand the complex search and rescue emergency plan process.

[0006] The virtual reality (VR) presentation provides an immersive experience for viewers through a fully immersive virtual environment where they can interact and gain in-depth insights into the details of search and rescue emergency operations. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a display method and device based on augmented reality. By superimposing virtual search and rescue information on a real terrain sand table environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of search and rescue emergency plans.

[0008] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses a display method based on augmented reality, the method comprising:

[0009] S1, obtaining real scene image information;

[0010] S2, performing calculation processing on the real scene image information to obtain target scene information;

[0011] S3, analyzing and processing the real scene image information and the target scene information to obtain target display information.

[0012] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the calculating and processing the real scene image information to obtain the target scene information includes:

[0013] S21, preprocessing the real scene image information to obtain preprocessed image information;

[0014] S22, analyzing and processing the pre-processed image information to obtain first angle information;

[0015] S23, calculating and processing the first angle information to obtain rotation angle information and offset information;

[0016] S24, converting the rotation angle information and the offset information to obtain target scene information.

[0017] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the analyzing and processing the pre-processed image information to obtain the first angle information includes:

[0018] S221, performing contour detection processing on the pre-processed image information to obtain contour information;

[0019] S222, filtering the contour information to obtain target contour information;

[0020] S223, performing calculation processing on the pre-processed image information and the target contour information to obtain first angle information.

[0021] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the calculating and processing the first angle information to obtain the rotation angle information and the offset information includes:

[0022] S231, obtaining second angle information and a scene angle training data set;

[0023] S232, performing training processing on the scene angle training data set to obtain rotation angle offset information and offset offset information;

[0024] S233, using a scene angle calculation model, calculating and processing the first angle information and the second angle information to obtain scene matrix information;

[0025] Wherein, the scene angle calculation model is:

[0026]

[0027] Where H is the scene matrix information, H i is the i-th scene matrix in the scene matrix information, the H i is a 3*3 matrix, (JDYX i,j ,JDYY i,j ) is the jth angle coordinate information corresponding to the i-th angle information in the first angle information, (JDEX i,j ,JDEYi i,j ) is the j-th angle coordinate information corresponding to the second angle information, and N is the number of angle information in the first angle information.

[0028] S234, calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain rotation angle information and offset information.

[0029] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain the rotation angle information and the offset information includes:

[0030] S2341, using a viewing angle offset calculation model, calculating and processing the scene matrix information to obtain pre-processing rotation angle information and pre-processing offset information;

[0031] Wherein, the viewing angle offset calculation model is:

[0032] (θx i1 ,θy i1 )=(δ1·arctan(H i1,32 ),δ2·arctan(H i1,31 ));

[0033]

[0034] Wherein, (θx, θy) is the pre-processed rotation angle information, (tx, ty) is the pre-processed offset information, (θx i1 ,θy i1 ) is the i1th rotation angle coordinate information in the pre-processed rotation angle information, (tx i1 ,ty i1 ) is the i1th offset coordinate information in the pre-processed offset information, H is the scene matrix information, H i1,23 , H i1,31 , H i1,32 and H i1,33are the values ​​of the 2nd row and 3rd column, the 3rd row and 1st column, the 3rd row and 2nd column, and the 3rd row and 3rd column of the i1th scene matrix in the scene matrix information, δ1, δ2, δ3, and δ4 are the first weight parameter, the second weight parameter, the third weight parameter, and the fourth weight parameter, respectively, 1≤i1≤N, and N is the number of scene matrices in the scene matrix information;

[0035] S2342, using a viewing angle offset calculation model, calculating and processing the rotation angle offset information, the offset offset information, the pre-processed rotation angle information, and the pre-processed offset information to obtain rotation angle information and offset information;

[0036] The viewing angle offset calculation model is:

[0037] θ′=δ5·θ+Δθ;

[0038] t′=δ6·t+Δt;

[0039] In the formula, θ′ is the rotation angle information, t′ is the offset information, θ and t are the preprocessed rotation angle information and the preprocessed offset information, Δθ and Δt are the rotation angle offset information and the offset offset information, δ5 and δ6 are the fifth weight parameter and the sixth weight parameter, respectively.

[0040] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the converting the rotation angle information and the offset information to obtain the target scene information includes:

[0041] S241, obtaining target coordinate information, target resolution information and target size information;

[0042] S242, using a scene conversion model, converting the target coordinate information, the target resolution information, the target size information, the rotation angle information, and the offset information to obtain target scene information;

[0043] Wherein, the scene conversion model is:

[0044]

[0045] Z = YS;

[0046] PYJ = θ′ + t′;

[0047] Where (X, Y, Z) is the corresponding coordinate information in the target scene information, PYJ is the corresponding plane offset angle information in the target scene information, X k1 and Y k2is the sum of the k1th X coordinate value and the k2th Y coordinate value in the corresponding coordinate information in the target scene information, X′ k1 and Y′ k2 is the k1th X coordinate value and the k2th Y coordinate value in the target coordinate information, YS is the Z coordinate value, XH and YH are the horizontal resolution and the vertical resolution in the target resolution information, L and D are the horizontal size information and the vertical size information in the target size information, M1 and M2 are the number of X coordinate values ​​and the number of Y coordinate values ​​in the target coordinate information, θ′ and t′ are the rotation angle information and the offset information.

[0048] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the analyzing and processing the real scene image information and the target scene information to obtain the target display information includes:

[0049] S31, obtaining virtual scene three-dimensional model information;

[0050] S32, traversing the virtual scene three-dimensional model information to obtain position information, direction information and scale information of the virtual scene three-dimensional model information;

[0051] S33, projecting the virtual scene three-dimensional model information onto the real scene image information and the target scene information according to the position information, the direction information and the scale information to obtain target display information.

[0052] A second aspect of an embodiment of the present invention discloses a display device based on augmented reality, the device comprising:

[0053] Acquisition model, used to obtain real scene image information;

[0054] A first calculation module is used to calculate and process the real scene image information to obtain target scene information;

[0055] The second computing module is used to analyze and process the target scene information to obtain target display information.

[0056] A third aspect of an embodiment of the present invention discloses another display device based on augmented reality, the device comprising:

[0057] processor;

[0058] a memory coupled to the processor and storing executable program code;

[0059] The processor calls the executable program code stored in the memory to execute part or all of the steps of the augmented reality-based display method disclosed in the first aspect of the embodiment of the present invention.

[0060] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps of the augmented reality-based display method disclosed in the first aspect of the embodiment of the present invention.

[0061] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0062] In the embodiment of the present invention, real scene image information is obtained; the real scene image information is calculated and processed to obtain target scene information; the target scene information is analyzed and processed to obtain target display information. It can be seen that the present application superimposes virtual search and rescue information on a real terrain sand table environment, and the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A flowchart of an augmented reality-based display method disclosed in an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the structure of an augmented reality display device disclosed in an embodiment of the present invention;

[0066] Figure 3 A schematic structural diagram of another display device based on augmented reality disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0069] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] The present invention discloses a display method and device based on augmented reality, by superimposing virtual search and rescue information on a real terrain sand table environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan. The following are detailed descriptions.

[0071] Embodiment 1

[0072] See also Figure 1 , Figure 1 is a flow chart of a display method based on augmented reality disclosed in an embodiment of the present invention. Figure 1 The augmented reality-based display method described is applied to an augmented reality-based display device, such as a local server or cloud server for display optimization management based on augmented reality, and the embodiments of the present invention do not limit this. Figure 1 As shown, the augmented reality-based display method may include the following operations:

[0073] S1, obtaining real scene image information;

[0074] It should be noted that the real scene image information is obtained by capturing the environment image in the user's field of view through the front camera equipped with the AR glasses, obtaining a real-time video stream, and arranging each frame in the real-time video stream in chronological order from small to large. The real scene image information includes multiple collected image information, that is, multiple continuous frames in the real-time video stream.

[0075] S2, performing calculation processing on the real scene image information to obtain target scene information;

[0076] S3, analyzing and processing the real scene image information and the target scene information to obtain target display information.

[0077] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0078] In an optional embodiment, the calculating and processing the real scene image information to obtain the target scene information includes:

[0079] S21, preprocessing the real scene image information to obtain preprocessed image information;

[0080] S22, analyzing and processing the pre-processed image information to obtain first angle information;

[0081] S23, calculating and processing the first angle information to obtain rotation angle information and offset information;

[0082] S24, converting the rotation angle information and the offset information to obtain target scene information.

[0083] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0084] In another optional embodiment, the preprocessing the real scene image information to obtain preprocessed image information includes:

[0085] S211, performing threshold processing on the real scene image information to obtain thresholded scene image information;

[0086] It should be noted that the above thresholding process is obtained through the thresholding calculation model;

[0087] Wherein, the threshold calculation model is:

[0088] YZ s1,s2 =δ7·U1 s1,s2 +δ8·U2 s1,s2 +δ9·U3 s1,s2 1≤s1≤N,1≤s2≤L;

[0089] δ7+δ8+δ9=1;

[0090] 0<δ7<1,0 <v8<1,0<δ9<1;

[0091] Where, YZ is the thresholded scene image information, YZ s1,s2 is the s2th pixel value of the s1th thresholded image information in the thresholded scene image information, U1 s1,s2 、U2 s1,s2 and U3 s1,s2 They are respectively the first channel value, the second channel value and the third channel value of the s2th pixel of the s1th acquired image information in the real scene image information, s1 is the number of the acquired image information in the real scene image information, L is the total number of pixels of the s1th acquired image information in the real scene image information, δ7, δ8, δ9 are the seventh weight parameter, the eighth weight parameter and the ninth weight parameter.

[0092] It should be noted that the first channel value, the second channel value and the third channel value represent one of the red channel value, the green channel value and the blue channel value. For example, if the first channel value is the red channel value and the second channel value is the blue channel value, then the third channel value is the green channel value. The specific setting is set by the user and is not specifically limited in the embodiment of the present invention.

[0093] It should be noted that the thresholding calculation model improves the accuracy and effect of thresholding through the reasonable weighting of the seventh weight parameter, the eighth weight parameter and the ninth weight parameter, which helps to remove noise while maintaining important features and enhance the available information of the image. At the same time, the weight parameters can be automatically adjusted according to actual needs, the image processing process can be automatically optimized, and the overall performance can be improved.

[0094] S212, performing denoising processing on the thresholded scene image information to obtain denoised image information;

[0095] It should be noted that the above denoising process is performed by a Gaussian filter, or other algorithms for denoising the image, which is not specifically limited in the embodiment of the present invention.

[0096] S213, performing edge enhancement processing on the denoised image information to obtain pre-processed image information.

[0097] It should be noted that the above-mentioned edge enhancement processing is performed by a Sobel operator, or other algorithms capable of performing edge enhancement processing, which is not specifically limited in the embodiment of the present invention.

[0098] It should be noted that, in the embodiment of the present invention, edge enhancement and contour detection are performed to identify and process the electronic sand table in the denoised image information. When AR glasses are used, the electronic sand table captured by the camera in the AR glasses is converted into a three-dimensional space scene, and the search and rescue plan model information (virtual scene three-dimensional model information) is projected in the three-dimensional space scene. The map data, topographic map and environmental features of the search and rescue area are then loaded and displayed in real time on the electronic sand table, thereby realizing the superposition of the virtual information of the search and rescue plan and the real scene information of the electronic sand table.

[0099] It should be noted that the electronic sand table integrates the Geographic Information System (GIS), which can load and display the map data, topographic maps and environmental features of the search and rescue area in real time, and synchronize the three-dimensional model information of the virtual scenes involved in the search and rescue to the display software on the AR glasses (used to display the display result information) to provide data support for it.

[0100] It should be noted that the electronic sandbox is a device that can display map data, topographic maps and environmental features of the search and rescue area. Its shape is rectangular, and its specific device type or information is not specifically limited in the embodiments of the present invention.

[0101] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0102] In yet another optional embodiment, the analyzing and processing the pre-processed image information to obtain the first angle information includes:

[0103] S221, performing contour detection processing on the pre-processed image information to obtain contour information;

[0104] It should be noted that the above-mentioned contour detection processing is to detect the edge of each preprocessed collected image information in the preprocessed image information through the Canny edge detection algorithm, and can also be through other edge detection processing algorithms, which are not limited in the embodiments of the present invention.

[0105] S222, filtering the contour information to obtain target contour information;

[0106] It should be noted that the above-mentioned screening process is performed by an algorithm set by a user, or by other algorithms capable of screening contour information, which is not specifically limited in the embodiment of the present invention.

[0107] S223, performing calculation processing on the pre-processed image information and the target contour information to obtain first angle information.

[0108] It should be noted that the above calculation processing is performed by polygonal approximation algorithm recognition. Exemplarily, if the electronic sand table is a long rectangle, the four vertices of the rectangle in the image are identified by polygonal approximation algorithm recognition, and the coordinate information of the four fixed points (including the first angle coordinate, the second angle coordinate, the third angle coordinate and the fourth angle coordinate) are recorded as angle information; wherein the first angle information includes calculating and processing the electronic sand table contour information in the target contour information corresponding to all preprocessed acquisition image information in the preprocessed image information and all preprocessed acquisition image information, and obtaining the angle information corresponding to each preprocessed acquisition image information.

[0109] It should be noted that the coordinate information in S223 and S231 is the coordinate in a two-dimensional plane coordinate system, that is, the lower left corner vertex of the preprocessed acquired image information in the preprocessed image information is taken as the origin, the direction from the lower left corner to the lower right corner is the positive direction of the X-axis, and the direction from the lower left corner to the upper left corner is the positive direction of the Y-axis. Each pixel point in the image is in the first quadrant of the coordinate system, and each pixel point corresponds to a coordinate point.

[0110] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0111] In an optional embodiment, the calculating and processing the first angle information to obtain the rotation angle information and the offset information includes:

[0112] S231, obtaining second angle information and a scene angle training data set;

[0113] It should be noted that the above-mentioned second angle information is the theoretical angle information of the electronic sand table in the pre-processed image information. For example, the vertex coordinates of the lower left corner of the theoretical angle information of the electronic sand table in the pre-processed image information are (150, 250), and the resolution of the electronic sand table is 1920×1080. The theoretical angle information is the lower left corner (150, 250), the lower right corner (150+1919, 250), the upper left corner (150, 250+1079), and the upper right corner (150+1919, 250+1079).

[0114] It should be noted that the scene angle training data set is a data set used to train the rotation angle information and offset information of the electronic sandbox in the real scene image information, and finally obtain the rotation angle offset information and the offset offset information. The data set can be obtained through previous test data or obtained through a public database, and the embodiment of the present invention does not make specific limitations.

[0115] S232, performing training processing on the scene angle training data set to obtain rotation angle offset information and offset offset information;

[0116] S233, using a scene angle calculation model, calculating and processing the first angle information and the second angle information to obtain scene matrix information;

[0117] Wherein, the scene angle calculation model is:

[0118]

[0119] Where H is the scene matrix information, H i is the i-th scene matrix in the scene matrix information, the H i is a 3*3 matrix, (JDYX i,j ,JDYYi i,j ) is the jth angle coordinate information corresponding to the i-th angle information in the first angle information, (JDEX i,j ,JDEYi i,j ) is the j-th angle coordinate information corresponding to the second angle information, and N is the number of angle information in the first angle information.

[0120] S234, calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain rotation angle information and offset information.

[0121] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0122] In an optional embodiment, training processing is performed on the scene angle training data set to obtain rotation angle offset information and offset offset information, including:

[0123] It should be noted that the scene angle training data set is trained to obtain the rotation angle offset information and the offset offset information, which can also be obtained by training a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0124] S2321, obtaining the rotation angle increment value and the offset increment value;

[0125] S2322, using a preset angle update model, calculating and processing the scene angle training data set to obtain a training rotation angle value and a training offset value;

[0126] It should be noted that the above-mentioned calculation process can be obtained by calculation in a calculation process mode set by the user, and the embodiment of the present invention does not make any specific limitation.

[0127] It should be noted that the above-mentioned preset angle update model can be a linear update model, a rotation matrix model, a convolutional neural network (CNN) or a recurrent neural network (RNN), and the specific embodiment of the present invention is not limited thereto.

[0128] S2323, obtaining a true rotation angle value and a true offset value corresponding to the scene angle training data set;

[0129] S2324, using an error calculation model, calculating and processing the rotation angle increment value, the offset increment value, the training rotation angle value, the training offset value, the true rotation angle value, and the true offset value to obtain an error rotation angle value and an error offset value;

[0130] Wherein, the error calculation model is:

[0131] θ w =θ z -θ x -θ l ;

[0132] t w =t z -t x -t l ;

[0133] In the formula, θ w and t w is the error rotation angle value and the error offset value, θ z and t z is the actual rotation angle value and the actual offset value, θ x and t x are the training rotation angle value and the training offset value, θ l and t l are the rotation angle increment value and the offset increment value;

[0134] By calculating the error rotation angle value and the error offset value, the system can evaluate the difference between the calculated results and the actual rotation angle and offset. This helps to optimize the system and adjust the calculation model to make it more accurate.

[0135] S2325, determining whether the error rotation angle value is less than a preset error rotation angle threshold, and obtaining a first error determination result;

[0136] When the first error judgment result is yes, execute S2326;

[0137] When the first error judgment result is no, obtaining a first scene angle training data set, determining that the first scene angle training data set is the scene angle training data set, and executing S2327;

[0138] S2326, determining whether the error offset value is less than a preset error offset threshold, and obtaining a second error determination result;

[0139] When the second error judgment result is no, obtaining a second scene angle training data set, determining that the second scene angle training data set is the scene angle training data set, and executing S2327;

[0140] When the second error judgment result is yes, execute S2328;

[0141] The first scene angle training data set contains more samples for rotation, and the second scene angle training data set contains more samples for offset. By judging whether the error rotation angle and the error offset are within the corresponding threshold range, the system can decide which training data set to use according to the different levels or sequences of errors (rotation angle error and offset error). This hierarchical processing helps to optimize the model for different problems and improve the effectiveness of training and adjustment. At the same time, the model can converge faster and more accurately.

[0142] It should be noted that the preset error rotation angle threshold and the preset error offset threshold may be set by a user or may be obtained based on historical data, which is not limited in the embodiment of the present invention.

[0143] Exemplarily, the value ranges of the preset error rotation angle threshold and the preset error offset threshold are both [0.01, 0.05].

[0144] S2327, using the parameter update model, updating the rotation angle increment value and the offset increment value to obtain an updated rotation angle increment value and an updated offset increment value, and determining that the updated rotation angle increment value is the rotation angle increment value, determining that the updated offset increment value is the offset increment value, and executing S2322;

[0145] Wherein, the parameter updating model is:

[0146] θ′ l ←θ l +θθ;

[0147] t′ l ←t l +tt;

[0148] In the formula, θ′ land t′ l is the updated rotation angle increment value and the updated offset increment value, θθ and tt are the preset rotation angle update step value and the preset offset update step value;

[0149] It should be noted that the preset rotation angle update step value and the preset offset update step value may be set by the user or obtained based on historical data, which is not limited in the embodiment of the present invention.

[0150] Exemplarily, the preset rotation angle update step value and the preset offset update step value are both 0.1.

[0151] S2328, determining that the rotation angle increment value is rotation angle offset information; determining that the offset increment value is offset offset information.

[0152] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0153] In an optional embodiment, the calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain the rotation angle information and the offset information includes:

[0154] S2341, using a viewing angle offset calculation model, calculating and processing the scene matrix information to obtain pre-processing rotation angle information and pre-processing offset information;

[0155] Wherein, the viewing angle offset calculation model is:

[0156] (θx i1 ,θy i1 )=(δ1·arctan(H i1,32 ),δ2·arctan(H i1,31 ));

[0157]

[0158] Wherein, (θx, θy) is the pre-processed rotation angle information, (tx, ty) is the pre-processed offset information, (θx i1 ,θy i1 ) is the i1th rotation angle coordinate information in the pre-processed rotation angle information, (tx i1 ,ty i1 ) is the i1th offset coordinate information in the pre-processed offset information, H is the scene matrix information, H i1,23 , Hi1,31 , H i1,32 and H i1,33 are the values ​​of the 2nd row and 3rd column, the 3rd row and 1st column, the 3rd row and 2nd column, and the 3rd row and 3rd column of the i1th scene matrix in the scene matrix information, δ1, δ2, δ3, and δ4 are the first weight parameter, the second weight parameter, the third weight parameter, and the fourth weight parameter, respectively, 1≤i1≤N, and N is the number of scene matrices in the scene matrix information;

[0159] It should be noted that the first weight parameter, the second weight parameter, the third weight parameter and the fourth weight parameter may be set by a user or may be obtained based on historical data, and the embodiment of the present invention does not specifically limit this.

[0160] It should be noted that the first weight parameter, the second weight parameter, the third weight parameter and the fourth weight parameter allow the rotation angle and the offset to be flexibly adjusted in different scenarios. If the rotation angle has a greater impact, the values ​​of δ1 and δ2 can be increased, and if the offset has a greater impact, the values ​​of δ3 and δ4 can be increased.

[0161] Through the above-mentioned perspective offset calculation model, more reliable geometric information can be extracted from the original scene data, thereby improving the credibility of the calculation and the ability to fit complex perspective changes in real scenes. At the same time, it can eliminate the influence of the absolute scale in the scene matrix and enhance the robustness of the calculation.

[0162] S2342, using a viewing angle offset calculation model, calculating and processing the rotation angle offset information, the offset offset information, the pre-processed rotation angle information, and the pre-processed offset information to obtain rotation angle information and offset information;

[0163] The viewing angle offset calculation model is:

[0164] θ′=δ5·θ+Δθ;

[0165] t′=δ6·t+Δt;

[0166] In the formula, θ′ is the rotation angle information, t′ is the offset information, θ and t are the preprocessed rotation angle information and the preprocessed offset information, Δθ and Δt are the rotation angle offset information and the offset offset information, δ5 and δ6 are the fifth weight parameter and the sixth weight parameter, respectively.

[0167] It should be noted that the fifth weight parameter and the sixth weight parameter may be set by a user or may be obtained based on historical data, and the embodiment of the present invention does not specifically limit this.

[0168] It should be noted that, through the combination of the fifth weight parameter, the sixth weight parameter, the rotation angle offset information and the offset offset information, the model has a stronger fault tolerance capability for abnormal values ​​or errors.

[0169] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0170] In an optional embodiment, the converting the rotation angle information and the offset information to obtain the target scene information includes:

[0171] S241, obtaining target coordinate information, target resolution information and target size information;

[0172] It should be noted that the target coordinate information is the coordinate information of the electronic sand table in the real scene image information in the plane two-dimensional coordinate system. The coordinate information here takes the vertex of the upper left corner of the electronic sand table in the real scene image information as the origin, and the direction from the upper left corner to the upper right corner is the positive direction of the X-axis, and the direction from the upper left corner to the lower left corner is the positive direction of the Y-axis. Each pixel point in the image is in the first quadrant of the coordinate system, and each pixel point corresponds to a coordinate point.

[0173] It should be noted that the target resolution information and the target size information are the actual resolution information and size information of the electronic sandbox.

[0174] S242, using a scene conversion model, converting the target coordinate information, the target resolution information, the target size information, the rotation angle information, and the offset information to obtain target scene information;

[0175] Wherein, the scene conversion model is:

[0176]

[0177] Z = YS;

[0178] PYJ = θ′ + t′;

[0179] Where (X, Y, Z) is the corresponding coordinate information in the target scene information, PYJ is the corresponding plane offset angle information in the target scene information, X k1 and Y k2 is the sum of the k1th X coordinate value and the k2th Y coordinate value in the corresponding coordinate information in the target scene information, X′ k1 and Y′ k2is the k1th X coordinate value and the k2th Y coordinate value in the target coordinate information, YS is the Z coordinate value, XH and YH are the horizontal resolution and the vertical resolution in the target resolution information, L and D are the horizontal size information and the vertical size information in the target size information, M1 and M2 are the number of X coordinate values ​​and the number of Y coordinate values ​​in the target coordinate information, θ′ and t′ are the rotation angle information and the offset information.

[0180] It should be noted that the Z coordinate value is set by the user or obtained based on historical data, and the embodiment of the present invention does not limit this.

[0181] It should be noted that, through the scene conversion model, the entire area of ​​the electronic sandbox is projected into a three-dimensional coordinate space according to the corresponding coordinate information and plane offset angle information in the target scene information.

[0182] It should be noted that the size of the electronic sandbox in the target scene information is completely consistent with that in the real scene image information.

[0183] It should be noted that after obtaining the above-mentioned target scene information, the edge of the target scene information can be adjusted to coincide with the edge of the electronic sandbox in the real scene image information through gestures in the AR glasses.

[0184] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0185] In an optional embodiment, the analyzing and processing the real scene image information and the target scene information to obtain the target display information includes:

[0186] S31, obtaining virtual scene three-dimensional model information;

[0187] It should be noted that the virtual scene three-dimensional model information is the three-dimensional model information of the rescue equipment (such as rescue aircraft, ships, etc.) in the search and rescue plan and its corresponding position information, direction information and scale information on the electronic sandbox.

[0188] S32, traversing the virtual scene three-dimensional model information to obtain position information, direction information and scale information of the virtual scene three-dimensional model information;

[0189] It should be noted that the above traversal can be implemented using Unity3D or other existing tools, and the embodiments of the present invention do not limit this.

[0190] S33, projecting the virtual scene three-dimensional model information onto the real scene image information and the target scene information according to the position information, the direction information and the scale information to obtain target display information.

[0191] It should be noted that the above-mentioned projection operation is obtained by projection using the display software on the AR glasses through simulation software. Through the projection operation, the AR glasses can simultaneously see the map data, topographic map and environmental features of the rescue site displayed on the electronic sand table in the real scene image information. At the same time, by projecting the virtual scene three-dimensional model information onto the target scene information, the status information of the virtual scene three-dimensional model information, such as the current position and direction of the rescue aircraft, can be seen in real time through the AR glasses in the target scene information above the electronic sand table.

[0192] It should be noted that the above-mentioned projection operation may be performed by simulation software set by the user, or may be obtained based on other existing projection technologies, which is not limited in the embodiment of the present invention.

[0193] It should be noted that the real-scene image information is based on the environmental image (video stream) arranged in chronological order within the user's field of view acquired by the AR glasses. After the embodiment of the present invention, the target display information obtained is also arranged in chronological order. The target display information is output and displayed in chronological order through the video display device in the AR glasses, and finally a dynamic output video stream will be obtained.

[0194] In the present invention, after the personnel participating in the simulation wear AR glasses and display information through the output target, they can interact with the rescue unit through gesture operations and voice commands.

[0195] It can be seen that by implementing the augmented reality-based display method described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0196] It should be noted that in the present invention, after obtaining the target display information, the target display information can also be maintained in the AR glasses through the scene-maintaining space persistence method. When the user leaves the original target display information, the virtual scene three-dimensional model information in the original target display information will be maintained at a specific position through the world anchor point. When the user returns to the target display information, the virtual scene three-dimensional model information content of the original target display information can be accurately restored. When the AR glasses are running, the current target display information will be saved in real time. When the user wears the glasses and returns to the target display information again, the AR glasses will match the current target display information. Over time, the AR glasses will build a spatial map of the environment it sees. Use spatial anchors to fix the position of the virtual image on the spatial map.

[0197] Embodiment 2

[0198] See also Figure 2 , Figure 2 is a schematic diagram of a display device based on augmented reality disclosed in an embodiment of the present invention. Figure 2 The augmented reality-based display device described is applied to an augmented reality-based display optimization system, such as a local server or cloud server for augmented reality-based display, etc., which is not limited in the embodiments of the present invention. Figure 2 As shown, the display device based on augmented reality includes:

[0199] Acquisition model 201, used to acquire real scene image information;

[0200] A first calculation module 202 is used to calculate and process the real scene image information to obtain target scene information;

[0201] The second calculation module 203 is used to analyze and process the target scene information to obtain target display information.

[0202] It can be seen that by implementing the augmented reality-based display device described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0203] Embodiment 3

[0204] See also Figure 3 , Figure 3 is a schematic diagram of the structure of another display device based on augmented reality disclosed in an embodiment of the present invention. Figure 3 The augmented reality-based display device described is applied to an augmented reality-based display optimization system, such as a local server or cloud server for augmented reality-based display, etc., which is not limited in the embodiments of the present invention. Figure 3As shown, the display device based on augmented reality includes:

[0205] Processor 301;

[0206] A memory 302 coupled to the processor 301 and storing executable program codes;

[0207] The processor 301 calls the executable program code stored in the memory 302 to execute part or all of the steps of the augmented reality-based display method described in Example 1.

[0208] It can be seen that by implementing the augmented reality-based display device described in the embodiment of the present invention, by superimposing virtual search and rescue information on a real terrain sandbox environment, the real world and virtual information can be seen at the same time, which is conducive to improving the collaborative efficiency and response speed of the search and rescue emergency plan.

[0209] Embodiment 4

[0210] An embodiment of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps of the augmented reality-based display method described in Example 1.

[0211] Embodiment 5

[0212] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps in the augmented reality-based display method described in Example 1.

[0213] The system embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative labor.

[0214] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0215] Finally, it should be noted that the display method and device based on augmented reality disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A display method based on augmented reality, characterized in that: The method comprises: S1, obtaining real scene image information; S2, performing calculation processing on the real scene image information to obtain target scene information; S3, analyzing and processing the real scene image information and the target scene information to obtain target display information.

2. The display method based on augmented reality according to claim 1, characterized in that: The calculating and processing the real scene image information to obtain the target scene information includes: S21, preprocessing the real scene image information to obtain preprocessed image information; S22, analyzing and processing the pre-processed image information to obtain first angle information; S23, calculating and processing the first angle information to obtain rotation angle information and offset information; S24, converting the rotation angle information and the offset information to obtain target scene information.

3. The display method based on augmented reality according to claim 2, characterized in that: The analyzing and processing the pre-processed image information to obtain the first angle information includes: S221, performing contour detection processing on the pre-processed image information to obtain contour information; S222, filtering the contour information to obtain target contour information; S223, performing calculation processing on the pre-processed image information and the target contour information to obtain first angle information.

4. The display method based on augmented reality according to claim 2, characterized in that: The calculating and processing the first angle information to obtain the rotation angle information and the offset information includes: S231, obtaining second angle information and a scene angle training data set; S232, performing training processing on the scene angle training data set to obtain rotation angle offset information and offset offset information; S233, using a scene angle calculation model, calculating and processing the first angle information and the second angle information to obtain scene matrix information; Wherein, the scene angle calculation model is: Where H is the scene matrix information, H i is the i-th scene matrix in the scene matrix information, the H i is a 3*3 matrix, (JDYX i,j ,JDYY i,j ) is the jth angle coordinate information corresponding to the i-th angle information in the first angle information, (JDEX i,j ,JDEYi i,j ) is the j-th angle coordinate information corresponding to the second angle information, and N is the number of angle information in the first angle information; S234, calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain rotation angle information and offset information.

5. The display method based on augmented reality according to claim 4, characterized in that: The calculating and processing the rotation angle offset information, the offset offset information, and the scene matrix information to obtain the rotation angle information and the offset information includes: S2341, using a viewing angle offset calculation model, calculating and processing the scene matrix information to obtain pre-processing rotation angle information and pre-processing offset information; Wherein, the viewing angle offset calculation model is: (θx i1 ,θy i1 )=(δ1·arctan(H i1,32 ),δ2·arctan(H i1,31 )); Wherein, (θx, θy) is the pre-processed rotation angle information, (tx, ty) is the pre-processed offset information, (θx i1 ,θy i1 ) is the i1th rotation angle coordinate information in the pre-processed rotation angle information, (tx i1 ,ty i1 ) is the i1th offset coordinate information in the pre-processed offset information, H is the scene matrix information, H i1,23 , H i1,31 , H i1,32 and H i1,33 are the values ​​of the 2nd row and 3rd column, the 3rd row and 1st column, the 3rd row and 2nd column, and the 3rd row and 3rd column of the i1th scene matrix in the scene matrix information, δ1, δ2, δ3, and δ4 are the first weight parameter, the second weight parameter, the third weight parameter, and the fourth weight parameter, respectively, 1≤i1≤N, and N is the number of scene matrices in the scene matrix information; S2342, using a viewing angle offset calculation model, calculating and processing the rotation angle offset information, the offset offset information, the pre-processed rotation angle information, and the pre-processed offset information to obtain rotation angle information and offset information; The viewing angle offset calculation model is: i ′ =δ5·θ+Δθ; t ′ =δ6·t+Δt; In the formula, θ ′ is the rotation angle information, t ′ is the offset information, θ and t are the preprocessed rotation angle information and the preprocessed offset information respectively, Δθ and Δt are the rotation angle offset information and the offset offset information respectively, δ5 and δ6 are the fifth weight parameter and the sixth weight parameter respectively.

6. The display method based on augmented reality according to claim 2, characterized in that: The converting process of the rotation angle information and the offset information to obtain the target scene information includes: S241, obtaining target coordinate information, target resolution information and target size information; S242, using a scene conversion model, converting the target coordinate information, the target resolution information, the target size information, the rotation angle information, and the offset information to obtain target scene information; Wherein, the scene conversion model is: Z = YS; PYJ = θ′ + t′; Where (X, Y, Z) is the corresponding coordinate information in the target scene information, PYJ is the corresponding plane offset angle information in the target scene information, X k1 and Y k2 is the sum of the k1th X coordinate value and the k2th Y coordinate value in the corresponding coordinate information in the target scene information, X′ k1 and Y′ k2 is the k1th X coordinate value and the k2th Y coordinate value in the target coordinate information, YS is the Z coordinate value, XH and YH are the horizontal resolution and the vertical resolution in the target resolution information, L and D are the horizontal size information and the vertical size information in the target size information, M1 and M2 are the number of X coordinate values ​​and the number of Y coordinate values ​​in the target coordinate information, θ ′ and t ′ The rotation angle information and the offset information.

7. The display method based on augmented reality according to claim 1, characterized in that: The analyzing and processing the real scene image information and the target scene information to obtain the target display information includes: S31, obtaining virtual scene three-dimensional model information; S32, traversing the virtual scene three-dimensional model information to obtain position information, direction information and scale information of the virtual scene three-dimensional model information; S33, projecting the virtual scene three-dimensional model information onto the real scene image information and the target scene information according to the position information, the direction information and the scale information to obtain target display information.

8. A display device based on augmented reality, characterized in that: The device comprises: Acquisition model, used to obtain real scene image information; A first calculation module is used to calculate and process the real scene image information to obtain target scene information; The second computing module is used to analyze and process the target scene information to obtain target display information.

9. A display device based on augmented reality, characterized in that: The device comprises: processor; a memory coupled to the processor and storing executable program code; The processor calls the executable program code stored in the memory to execute the augmented reality-based display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the augmented reality-based display method according to any one of claims 1 to 7.

Images