A target superimposition method and system for augmented reality

Abstract

The present application relates to the technical field of virtual-real fusion image processing, and particularly relates to a target superimposition method and system for augmented reality. The present application firstly simulates a virtual route target and superimposes and fuses the virtual route target into an initial real picture to guide navigation. Then, the present application analyzes the change degree of a current real picture to determine whether the virtual target and the current real picture need to be superimposed and fused for updating. If yes, the present application analyzes the similarity between the current real picture and a reference picture in local areas, and preferentially matches the local areas. Then, the present application evaluates the relative displacement of a device according to the matching results of the corner points in the local areas, superimposes and fuses the virtual target and the current real picture for updating, and obtains a more accurate superimposition and fusion result of the virtual target, thereby enhancing the immersion and accuracy of the user during use. Moreover, under the premise of ensuring the accuracy, the present application effectively improves the efficiency of AR operation, reduces the calculation delay, and reduces the loss of device performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual-real fusion image processing, in particular to a target superimposition method and system for augmented reality. BACKGROUND

[0002] Augmented reality is a new interactive technology that fuses real environment and virtual information. It simulates and superimposes virtual information that is difficult to directly perceive into a real scene through computer graphics, photoelectric display, multi-sensor technology, etc., so that users can interact with virtual objects in real time in the background of the real world. This technology not only realizes the organic integration of virtual elements and physical environment, but also provides an immersive experience beyond reality through three-dimensional registration and real-time tracking technology. Augmented reality needs to complete the fusion of virtual and real in three-dimensional space, and special display devices are needed to meet the fusion requirements. Currently, common augmented reality display devices can be roughly divided into three categories: wearable devices, mobile handheld display devices, and spatial augmented devices.

[0003] In the process of fusing virtual targets and real scenes, users often move the augmented reality device screen to observe more virtual information. For example, in an exhibition hall, the virtual projected target is large, and the device camera is difficult to shoot completely. In augmented reality navigation, the user's position changes in real time, and the augmented reality device screen also changes rapidly. The virtual navigation index needs to be updated and fused with the user to make the user experience more smooth and smooth. However, when the user moves quickly, the shooting screen changes greatly, and it is often difficult to match the relative position in the screen and re-fuse the virtual information.

[0004] To solve the problem that the rapid change of the device's shooting of the real world screen in the augmented reality process leads to displacement deviation in the superimposition and fusion of virtual target information, the prior art often identifies and matches the corners of the real world screen before and after the change, evaluates the relative displacement direction and distance, and adjusts the position of the virtual target in the changed real world screen. However, when the shooting angle changes rapidly, the real world screen changes due to the change of the lens position, and two images have their own corners, which makes the corner matching not accurate enough, resulting in low accuracy of subsequent operations and inaccurate superimposition and fusion results. SUMMARY

[0005] To solve the above technical problems, the purpose of the present application is to provide a target superimposition method and system for augmented reality.

[0006] According to the first aspect of the embodiment of the present application, a target superimposition method for augmented reality is provided, and the technical solution is as follows:

[0007] real-time acquisition of a real scene, initialization of a virtual target and initial superimposition and fusion of the virtual target with an initial real scene;

[0008] the last virtual target and real scene superimposition and fusion update result is recorded as a reference picture, and a difference degree between a current real scene and the reference picture is analyzed to determine whether virtual target and current real scene superimposition and fusion update is needed;

[0009] If so, a local similarity between local areas of the current real scene and the reference picture is analyzed, and a local consistent area between the current real scene and the reference picture is matched;

[0010] corners of the local consistent area in the current real scene and the reference picture are matched to obtain a relative displacement vector of the device;

[0011] According to the relative displacement vector, the superimposition and fusion update of the virtual target and the current real scene is performed.

[0012] In some embodiments of the present application, analyzing the difference degree between the current real scene and the reference picture comprises:

[0013] a first pixel difference value between all real scenes and their corresponding reference pictures is obtained to obtain a picture difference value matrix set and a current picture difference value matrix;

[0014] The average value of each picture difference value matrix element value in the picture difference value matrix set is calculated to obtain a maximum difference value matrix average value and a current picture difference value matrix average value;

[0015] The position distribution uniformity of the maximum value point in the current picture difference value matrix is analyzed, and the maximum difference value matrix average value and the current picture difference value matrix average value are combined to obtain the difference degree between the current real scene and the reference picture.

[0016] In some embodiments of the present application, analyzing the position distribution uniformity of the maximum value point in the current picture difference value matrix comprises:

[0017] The nearest distance between each maximum value point and its nearest maximum value point in the current picture difference value matrix is obtained, and the farthest distance between each maximum value point and its farthest maximum value point is obtained, the difference between the farthest distance and the nearest distance is calculated, and the number of diagonal line data in the current picture difference value matrix is combined to obtain the position distribution uniformity of the maximum value point in the current picture difference value matrix.

[0018] In some embodiments of the present application, determining whether virtual target and current real scene superimposition and fusion update is needed comprises:

[0019] A preset difference threshold;

[0020] determining whether the difference is greater than or equal to the difference threshold;

[0021] If yes, a virtual target and a current real picture are superimposed and fused to update.

[0022] In some embodiments of the present application, analyzing the local similarity between the local region between the current real picture and the reference picture comprises:

[0023] determining a local overlap mode in which the similar region between the current real picture and the reference picture is the most, to obtain an overlap region;

[0024] obtaining a pixel difference minimum point of the overlap region in the current real picture and the reference picture respectively;

[0025] In the current real picture and the reference picture respectively, the pixel difference minimum point is taken as the center to gradually expand the region, to obtain a corresponding expansion new point set in the current real picture and the reference picture;

[0026] Analyzing the second pixel difference and its distribution between the expansion new point set and the current real picture and the reference picture, to obtain the consistency degree of the expansion new point set;

[0027] According to the consistency degree, combining the difference, the local similarity between the local region between the current real picture and the reference picture is obtained.

[0028] In some embodiments of the present application, according to the consistency degree, combining the difference, the local similarity between the local region between the current real picture and the reference picture is obtained, comprising:

[0029] Based on the consistency degree, the consistency degree similarity between the pixel difference minimum point corresponding region before the current step region expansion and the expansion new point set after the current step expansion is analyzed;

[0030] Based on the difference, the difference degree similarity between the difference between the current real picture and the reference picture and the difference between the expansion new point set after the current step expansion and the overlap region is analyzed;

[0031] Combining the consistency degree similarity and the difference degree similarity, the expansion region similarity of the pixel difference minimum point corresponding region in the current real picture and the reference picture after the current step region expansion is obtained;

[0032] According to the expansion region similarity, the local similarity between the local region between the current real picture and the reference picture is obtained.

[0033] In some embodiments of the present application, the local similarity between the local region between the current real picture and the reference picture is obtained according to the expansion region similarity, comprising:

[0034] After each step of region expansion, the size of the expansion region similarity value corresponding to the region before and after the expansion is compared;

[0035] If the expansion region similarity value becomes larger, the current region expansion is retained, and the next step of region expansion is continued;

[0036] If the expansion region similarity value becomes smaller, the last step of region expansion is retained and the expansion is stopped, and the expansion region similarity corresponding to the last step of region expansion is recorded as the local similarity between the local region between the current real picture and the reference picture.

[0037] In some embodiments of the present application, the relative displacement vector of the device is obtained by matching the corner points of the local consistent region in the current real picture and the reference picture, comprising:

[0038] Obtaining the corner points of the current real picture and the reference picture;

[0039] Based on the corner points, using a corner point matching algorithm, the highest matching degree of the corner point pair of the local consistent region in the current real picture and the reference picture is matched, and the vector from the corner point position on the reference picture to the corner point position on the current real picture in the highest matching degree of the corner point pair is recorded as the displacement vector of the device;

[0040] According to the displacement vector, the relative displacement vector of the device is obtained in combination with the local similarity corresponding to the local consistent region.

[0041] According to a second aspect of an embodiment of the present application, a target superimposition system for augmented reality is provided, comprising a memory and a processor, wherein:

[0042] The memory is configured to store program code;

[0043] The processor is configured to read the program code stored in the memory and execute the method of the first aspect of the present application.

[0044] In some embodiments of the present application, the processor comprises:

[0045] A picture acquisition and preliminary superimposition fusion module is configured to acquire a real picture in real time, initialize a virtual target and perform preliminary superimposition fusion of the virtual target and an initial real picture;

[0046] The current real scene change degree analysis module is used for recording the last virtual target and the real scene superimposed and fused updated picture as a reference picture, analyzing the difference between the current real scene and the reference picture, and judging whether the virtual target and the current real scene superimposed and fused update is needed or not.

[0047] The superimposed and fused update module is used for analyzing the local similarity between the local area of the current real scene and the reference picture, matching the local consistent area between the current real scene and the reference picture, further matching the corner point of the local consistent area in the current real scene and the reference picture, obtaining the relative displacement vector of the device, and performing the superimposed and fused update of the virtual target and the current real scene according to the relative displacement vector.

[0048] Compared with the prior art, the target superimposition method and system for augmented reality provided by the application has the following beneficial effects:

[0049] The application firstly obtains the initial position and target position of the user, simulates the virtual route target, and superimposes and fuses it into the initial real scene picture of the user to guide navigation. During the movement of the camera of the user device, the obtained real scene changes, the change degree of the current real scene is evaluated according to the change of the real scene, so as to determine whether the superimposed and fused update of the virtual target and the current real scene is needed or not. For the small picture change, the superimposed and fused update is not performed, the efficiency of the AR algorithm operation can be effectively improved, the calculation delay can be reduced, and the performance loss of the device can be reduced. For the real scene that needs to be superimposed and fused, the local similarity between the same real local area of the current real scene and the reference picture is matched, the local area is preferentially matched, the relative displacement of the device is evaluated according to the matching result of the corner point in each local area, the position change and superimposed and fused update of the virtual target are quickly performed, a more accurate superimposed and fused result of the virtual target is obtained, the immersion and accuracy of the user during use are enhanced, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art and the advantages thereof, hereinafter, a brief introduction will be given to the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0051] Figure 1 The basic flowchart of a target superimposition method for augmented reality provided by an embodiment of the application is shown in the figure.

[0052] Figure 2 An example diagram of a same real partial area and virtual path relative device displacement substantially consistent between a current real picture and a reference picture according to an embodiment of the present application;

[0053] Figure 3 An example diagram of an overlapping area formed from relative movement of a diagonal point between a current real picture and a reference picture according to an embodiment of the present application;

[0054] Figure 4 An example diagram of a device displacement vector according to a highest matching degree of a diagonal point pair according to an embodiment of the present application;

[0055] Figure 5 A basic component diagram of a target superimposition system for augmented reality according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined inventive purpose, the following describes in detail the specific implementation, structure, features and effects of a target superimposition method and system for augmented reality according to the present application, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The use of the terms "including", "containing" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, an element defined by the phrase "including a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element. The relative terms "first" and "second" and the like are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations.

[0058] The scenario addressed by the present application is that in the case of frequent user movement, such as navigation in a library or a shopping mall, the relative position between the virtual target and the real picture is prone to deviation, the displacement distance is large, and the update has a certain delay, which affects the user experience.

[0059] Therefore, the main purpose of the present application is to solve the deviation phenomenon of the relative position of the superimposed fusion result of the virtual target when the real picture changes, caused by the quick movement of the user during the use of the device, the quick change of the camera view angle of the device, the large and quick change of the real picture, and the large and quick change of the real picture.

[0060] In order to achieve the above purpose of the present application, the present application further compares the real pictures obtained before and after the movement of the device, and obtains the corresponding positions of the same real picture in the two images before and after the movement of the device according to the difference results of different local areas, so as to more accurately match the corner points in the local area, obtain the relative displacement degree before and after the movement, and more accurately superimpose and fuse the virtual target.

[0061] The specific scheme of the target superimposition method for augmented reality provided by the present application will be described in detail below with reference to the accompanying drawings.

[0062] Please refer to Figure 1 , which shows the basic flow of the target superimposition method for augmented reality provided by an embodiment of the present application.

[0063] As Figure 1 shown, the target superimposition method for augmented reality provided by an embodiment of the present application specifically comprises:

[0064] S100: Real-time acquisition of a real picture, initialization of a virtual target, and preliminary superimposition and fusion of the virtual target with an initial real picture.

[0065] Real-time acquisition of a real picture, initialization of a virtual target, and preliminary superimposition and fusion of the virtual target with an initial real picture. Specifically, first, an initial real picture at an initial position of a user is acquired by shooting with a designated camera; then, according to the initial position of the user, the initial real picture acquired by the camera of the device, and a target position set by the user, a three-dimensional space of the initial scene where the user is located is modeled in combination with AR navigation technology (Augmented Reality Navigation) to acquire real environment images around the user shot by the camera of the device used by the user; and an AR algorithm (Autoregressive algorithm) is used to initialize the virtual target and accurately superimpose a virtual navigation mark in the initial real picture to guide the user to move towards the target position set by the user. During the movement of the user, a real picture shot by the camera of the device used by the user is acquired in real time.

[0066] At this point, the initial superimposition and fusion result of the virtual target is obtained, and the real picture is updated in real time.

[0067] Since the AR algorithm needs to perform a large number of complex operations, the requirements for time and computing power are large, and it is time-consuming and consumes device performance to generate and superimpose and fuse a virtual target in real time according to a real picture using the AR algorithm. In actual movement, small displacement behaviors are common, and the same real area exists in the real pictures before and after movement, so the relative displacement of the device can be obtained to quickly change the position of the virtual target and superimpose and fuse. Therefore, during the adjustment interval of the AR algorithm, the superimposition and fusion of the virtual target can be updated according to the existing superimposition result and the change of the real picture to reduce performance consumption and calculation delay on the premise of ensuring accuracy. In the embodiments of the present application, specific steps S200 to S500 are included.

[0068] S200: record the picture obtained after the last virtual target and real picture superimposition and fusion update as a reference picture, analyze the difference between the current real picture and the reference picture, and determine whether the virtual target and the current real picture superimposition and fusion update is needed.

[0069] First, record the picture obtained after the last virtual target and real picture superimposition and fusion update operation as a reference picture, and record the current real picture obtained in real time as a current real picture. When the user moves or rotates the AR device, the camera on the device moves, and the real picture obtained changes accordingly, so there is a high probability that there is a large difference between the two real pictures before and after movement. Therefore, the difference between the current real picture and the reference picture can be analyzed to determine whether the virtual target and the current real picture superimposition and fusion update is needed, that is, to evaluate whether the current real picture has changed significantly to the extent that the virtual target and the current real picture superimposition and fusion update operation is needed.

[0070] Further, the difference between the current real picture and the reference picture is analyzed, including:

[0071] First, obtain the first pixel difference between all real pictures and their corresponding reference pictures to obtain a picture difference matrix set and a current picture difference matrix. Specifically, record the real picture obtained at a certain time every certain period of time (which can be defined by the user, and in this example, once per second), calculate the first pixel difference (it is stipulated that the obtained pixel difference is an absolute value, which will not be described below) between the real picture and the reference picture at that moment using the frame difference method to obtain a picture difference matrix; assuming that a total of N real pictures are taken up to the current real picture, the set of picture difference matrices corresponding to each real picture up to the current real picture is recorded as , where represents the picture difference matrix set, represents the picture difference matrix corresponding to the i-th real picture, ​​represents the number of real pictures obtained by shooting up to the current real picture, and the current picture difference matrix is .

[0072] Then, the element value in the picture difference matrix represents the difference between two pictures in the same region, and the larger the element value, the greater the pixel difference between the two pictures at that point. Therefore, in some embodiments of the present application, the mean value of the element value of each picture difference matrix in the picture difference matrix set is calculated (denoted as ), the maximum difference matrix mean value (denoted as ) is obtained, and the mean value of the element value of the current picture difference matrix (denoted as ) is obtained.

[0073] In addition, the maximum value point in the picture difference matrix is the point with the largest difference in the local region, and in the two pictures with large differences, each local region has a large difference, and the maximum value point has a large probability of being more and uniformly distributed. Therefore, in some embodiments of the present application, the position distribution uniformity of the maximum value point in the current picture difference matrix is analyzed. Specifically, the maximum value point in the current picture difference matrix is taken, and the number is denoted as ; the nearest distance between each maximum value point in the current picture difference matrix and its nearest maximum value point (denoted as ) is obtained, and the farthest distance between each maximum value point and its farthest maximum value point (denoted as ) is obtained, the difference between the farthest distance and the nearest distance is calculated, denoted as , wherein represents the difference between the farthest distance and the nearest distance corresponding to the th maximum value point in the current picture difference matrix , and represents the farthest distance corresponding to the th maximum value point in the current picture difference matrix , and represents the nearest distance corresponding to the th maximum value point in the current picture difference matrix ; the number of data on the diagonal line of the current picture difference matrix is ; the difference between the farthest distance and the nearest distance corresponding to the th maximum value point and half of the number of data on the diagonal line of the current picture difference matrix are combined, and the position distribution uniformity of the maximum value point in the current picture difference matrix is obtained by traversing all the maximum value points in the current picture difference matrix. ​

[0074] Finally, the difference degree between the current real-time picture and the reference picture is obtained by combining the position distribution uniformity of the maximum value points in the current picture difference matrix, the maximum difference matrix mean value and the mean value of the current picture difference matrix. Then the difference degree between the current real-time picture and the reference picture is calculated according to the following formula:

[0075]

[0076] In the formula, Diff represents the difference degree between the current real-time picture and the reference picture; Mean represents the mean value of the current picture difference matrix element value; MaxMean represents the maximum difference matrix mean value; MaxNum represents the number of maximum value points in the current real-time picture; MaxDiff represents the difference between the farthest distance and the nearest distance corresponding to the maximum value points in the current real-time picture; Diag represents the number of diagonal line data of the current picture difference matrix; and Num represents the number of maximum value points in the current real-time picture.

[0077]

[0078] ​​​​​​​​​​​​​​​​​​​​​​​The greater the value, the greater the difference between the current real picture and the reference picture, and the more the virtual target and the current real picture should be superimposed and fused. Therefore, after obtaining the difference between the current real picture and the reference picture, it is judged whether the virtual target and the current real picture need to be superimposed and fused according to the difference. Specifically, a difference threshold is preset, and the value can be 0.65; it is judged whether the difference is greater than or equal to the difference threshold; if yes, it is considered that the difference between the current real picture and the reference picture is large, and the virtual target and the current real picture need to be superimposed and fused; if no, it is considered that the difference between the current real picture and the reference picture is small, and the virtual target and the current real picture do not need to be superimposed and fused.

[0079] S300: Analyzing the local similarity of the local area between the current real picture and the reference picture, and matching the local consistent area between the current real picture and the reference picture.

[0080] According to the above step S200, if it is judged that the difference between the current real picture and the reference picture is large, the virtual target and the current real picture need to be superimposed and fused. Although there is a large difference between the current real picture and the reference picture, because the two pictures are obtained by the displacement of the camera in the same real scene, there is a high probability that some local areas of the same real picture are in different positions of the two pictures, as shown in FIG. 1, the relative change of the position can represent the change of the virtual target relative to the camera, so it is necessary to identify and match the local areas of the same real picture in the current real picture and the reference picture. Figure 2

[0081] Based on the above analysis, in the embodiments of the present application, when the virtual target and the current real picture are superimposed and fused, the local similarity of the local area between the current real picture and the reference picture is analyzed first, and the local consistent area between the current real picture and the reference picture is matched. Further, it includes:

[0082] First, the local overlap mode with the most similar area between the current real picture and the reference picture is determined, and the overlapping area is obtained. Specifically, because the shooting device is displaced, there is a consistent area and an area unique to each of the current real picture and the reference picture, so the local overlap mode with the most local consistent area between the current real picture and the reference picture is determined first. The diagonal points of the current real picture and the reference picture are moved relative to each other, as shown in FIG. 2, the right lower point of the a image (reference picture) and the left upper point of the b image (current real picture) are started, and the b image is translated by a unit distance upward and leftward, so as to obtain the overlapping area of the a image and the b image at each time. Figure 3

[0083] ​​​​Furthermore, the minimum pixel difference points of the overlapping region within the current real-world image and the reference image are obtained. Specifically, the number of minimum pixel difference points and the average minimum pixel difference points of the overlapping region between the current real-world image and the reference image are obtained. The overlapping method corresponding to the minimum average minimum pixel difference point is selected as the local overlapping method with the most similar regions. When the average minimum pixel difference point is the same, the overlapping method corresponding to the maximum number of minimum pixel difference points is selected as the local overlapping method with the most similar regions. The overlapping region corresponding to the local overlapping method is obtained, and the minimum pixel difference points of the overlapping region within the current real-world image and the reference image are obtained.

[0084] Then, in both the current real-world image and the reference image, the region is gradually expanded centered on the point with the minimum pixel difference, resulting in a set of newly added points within both the current real-world image and the reference image. Specifically, based on the minimum pixel difference points within the overlapping region selected above, the same real-world region containing each minimum pixel difference point is expanded and divided. Taking a specific minimum pixel difference point as an example, the neighborhood range is gradually expanded outward from the corresponding points in both the current real-world image and the reference image, resulting in a set of newly added points within both the current real-world image and the reference image.

[0085] Furthermore, the smaller the pixel difference between different locations in the current real-world image and the reference image, the higher the similarity at that point, and the more likely they are points from the same local real-world region. Similarly, the smaller the pixel difference around the point with the minimum pixel difference and the greater the clustering of these points, the higher the similarity between the current real-world image and the reference image in that region, and the larger the area of ​​that region, and the more likely it is that the region is from the same local real-world region. Therefore, in some embodiments of the present invention, the consistency of the expanded new point set is obtained by analyzing the second pixel difference and its distribution between the current real-world image and the reference image. Specifically, taking the current step region expansion as an example, the number of expanded new points within the expanded new point set obtained from the current step region expansion is obtained. And obtain the second pixel difference (denoted as ) between the current real-world image and the reference image for all expanded new points in the expanded new point set. The total pixel difference of all newly added points within the expanded point set is calculated and denoted as . Further, obtain the maximum and minimum values ​​of the second pixel difference corresponding to the newly added points in the current step of expansion, and calculate the distance between the newly added points corresponding to the maximum and minimum values ​​of the second pixel difference (denoted as ). ), and the difference between the maximum and minimum values ​​of the second pixel difference (denoted as ). Therefore, the formula for calculating the consistency of the newly added point set in the expansion is as follows:

[0086] ;

[0087] In the formula, Indicates the degree of consistency of the newly added point set during expansion; This represents the total pixel difference among all newly added points within the expanded point set. This represents the distance between the newly added points corresponding to the maximum and minimum values ​​of the second pixel difference in the set of expanded new points. This represents the difference between the maximum and minimum values ​​of the second pixel difference in the newly added point set.

[0088] The total pixel difference of the expanded new point set is used to visually represent the size of the pixel difference in the expanded new point set. The smaller the value, the lower the overall pixel difference of the expanded new point set and the higher the similarity. and The ratio represents the distance between the points with the largest and smallest pixel differences in the newly added points set, and the magnitude of their difference. A larger ratio indicates a greater distance between the two newly added points and a smaller difference between them, suggesting a more uniform and consistent set of differences. In summary, the consistency of the newly added point set... The larger the value, the lower the overall pixel difference value of the expanded new point set, and the higher the consistency, indicating that the similarity of the regions where they are located is higher.

[0089] Using the same method, the consistency of the original point set (the point set formed by the regions corresponding to the minimum pixel differences before the current step region expansion) is calculated and denoted as . .

[0090] Finally, based on the degree of consistency and combined with the degree of difference, the local similarity of local areas between the current real-world image and the reference image is obtained. Furthermore, this includes:

[0091] First, based on the degree of consistency, we analyze the similarity between the regions corresponding to the minimum pixel difference points before the current step of region expansion and the newly added point set in the current step, that is, the degree of consistency of the point sets formed by the regions corresponding to the minimum pixel difference points before the current step of region expansion. The degree of consistency between the current step expansion and the newly added point set. The degree of similarity between them.

[0092] Simultaneously, based on the difference degree, the similarity between the difference degree between the current real image and the reference image and the difference degree between the newly added point set in the current step of expansion in the overlapping area is analyzed. Specifically, the difference degree between the local areas formed after the current area expansion is completed by the pixel difference minimum point in the current real image and the reference image, respectively, is calculated using the same method as in step S200, and is denoted as... .

[0093] Then, the expansion region similarity of the pixel difference minimum point corresponding region in the current real picture and the reference picture is obtained by combining the consistency degree similarity and the difference degree similarity, and the expansion region similarity calculation formula of the pixel difference minimum point corresponding region in the current real picture and the reference picture after the current step region expansion is constructed as follows:

[0094] ;

[0095] In the formula, represents the expansion region similarity of the pixel difference minimum point corresponding region in the current real picture and the reference picture after the current step region expansion; represents the difference degree between the current real picture and the reference picture; represents the difference degree between the local regions formed in the current real picture and the reference picture after the current region expansion of the pixel difference minimum point; represents the consistency degree of the expansion new point set; represents the consistency degree of the point set formed by the pixel difference minimum point corresponding region before the current step region expansion; represents the absolute value.

[0096] The greater the value is, the greater the difference degree between the current real picture and the reference picture is. The smaller the value is, the smaller the difference degree of the local region in the whole picture is, and the higher the similarity of the corresponding local region in the current real picture and the reference picture is. The greater the value is, the higher the similarity of the expansion new point set region in the current real picture and the reference picture is, and the smaller the difference The greater the value is, the greater the difference degree between the current real picture and the reference picture is. The smaller the value is, the more similar the difference amount of the expansion new point set region relative to the original region before expansion is, and the greater the expansion region is and the lower the difference in the region is, and the higher the similarity of the local region as a whole is.

[0097] Further, the local similarity of the local region between the current real picture and the reference picture is obtained according to the expansion region similarity. Specifically, after each step region expansion, the size of the expansion region similarity value corresponding to the region before and after the expansion is compared; if the expansion region similarity value becomes larger, the current step region expansion is retained, and the next step region expansion is continued (the expansion is continued to the next round of neighborhood); if the expansion region similarity value becomes smaller, the last step region expansion is retained and the expansion is stopped, and the expansion region similarity corresponding to the last step region expansion is recorded as the local similarity of the local region between the current real picture and the reference picture.

[0098] Similarly, the minimum value points of all pixels in the overlapping area between the current real picture and the reference picture are regionally expanded, and after the region expansion, the final expansion area corresponding to each pixel difference minimum value point is obtained, which is recorded as the local consistent area between the current real picture and the reference picture, and the local similarity corresponding to each local consistent area is obtained.

[0099] S400: The corner points of the matched local consistent area in the current real picture and the reference picture are matched to obtain the relative displacement vector of the device.

[0100] The corner points of the matched local consistent area in the current real picture and the reference picture are matched to obtain the relative displacement vector of the device. Further, comprising:

[0101] Firstly, the corner points of the current real picture and the reference picture are obtained. Specifically, the corner point position of the current real picture and the reference picture is identified and marked using a corner point detection algorithm.

[0102] Then, based on the corner points, the corner point pair with the highest matching degree of the local consistent area in the current real picture and the reference picture is matched using a corner point matching algorithm, and the vector from the position of the corner point on the reference picture (the corresponding corner point of the reference picture in the corner point pair with the highest matching degree) to the position of the corner point on the current real picture (the corresponding corner point of the current real picture in the corner point pair with the highest matching degree) is recorded as the displacement vector of the device, as shown in formula (1). Figure 4

[0103] Finally, according to the displacement vector, the relative displacement vector of the device is obtained in combination with the local similarity corresponding to the local consistent area. Specifically, assuming that a total of N local consistent areas are obtained, the local similarity of each local consistent area is recorded as S1, S2, …, SN, the sum of the local similarities of all N local consistent areas is recorded as S, and the relative displacement vector of the device is obtained in combination with the displacement vector and the sum of the local similarities, as shown in formula (2).

[0104] ;

[0105] In the formula, V represents the relative displacement vector of the device;

[0106] ​​​​​​​​​​​​The relative value of the local similarity The displacement vector of the device is weighted to obtain a relative displacement vector of the device, and the larger the value is, the more obvious the movement of the device is.

[0107] S500: According to the relative displacement vector, the superimposition and fusion update of the virtual target and the current real picture is performed.

[0108] According to the relative displacement vector, the superimposition and fusion update of the virtual target and the current real picture is performed, that is, the relative position of the virtual target in the current real picture is displaced by the relative displacement vector and then the superimposition and fusion update operation is performed, to obtain an updated superimposition and fusion effect, guiding the user to move to the target position set by the user.

[0109] Based on the same inventive concept as the above method, the embodiment also provides a target superimposition system for augmented reality.

[0110] Please refer to Figure 5 which shows the basic components of a target superimposition system for augmented reality provided by an embodiment of the present application.

[0111] As shown in Figure 5 , a target superimposition system for augmented reality comprises a memory 10 and a processor 20, wherein:

[0112] The memory 10 is used to store program codes;

[0113] The processor 20 is used to read the program codes stored in the memory 10, and perform real-time acquisition of a real picture, initialization of a virtual target and preliminary superimposition and fusion of the virtual target with an initial real picture; the picture obtained after the superimposition and fusion update of the virtual target with the real picture last time is taken as a reference picture, the difference degree between the current real picture and the reference picture is analyzed, and it is judged whether the superimposition and fusion update of the virtual target with the current real picture is needed; if yes, the local similarity of local areas between the current real picture and the reference picture is analyzed, the local consistent areas between the current real picture and the reference picture are matched; the corner points of the matched local consistent areas in the current real picture and the reference picture are obtained to obtain a relative displacement vector of the device; and the superimposition and fusion update of the virtual target with the current real picture is performed according to the relative displacement vector.

[0114] Further, the processor 20 comprises a picture acquisition and preliminary superimposition and fusion module 21, a current real picture change degree analysis module 22 and a superimposition and fusion update module 23, wherein:

[0115] The picture acquisition and preliminary superimposition and fusion module 21 is used to acquire a real picture in real time, initialize a virtual target and perform preliminary superimposition and fusion of the virtual target with an initial real picture;

[0116] The current real scene change degree analysis module 22 is used for taking the last virtual target and real scene superimposition fusion updated picture as a reference picture, analyzing the difference between the current real scene and the reference picture, and judging whether the virtual target and the current real scene superimposition fusion update is needed;

[0117] The superimposition fusion update module 23 is used for analyzing the local similarity between the local area of the current real scene and the reference picture, matching the local consistent area between the current real scene and the reference picture, further matching the corner point of the local consistent area in the current real scene and the reference picture, obtaining the relative displacement vector of the device, and performing the superimposition fusion update of the virtual target and the current real scene according to the relative displacement vector.

[0118] It should be noted that the above-mentioned embodiment sequence of the application is only for description, and does not represent the advantages and disadvantages of the embodiment. The process depicted in the drawing does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.

[0119] Each embodiment in the specification is described in a progressive manner, and the same and similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments.

Claims

1. A target superimposition method for augmented reality, characterized by, The method comprises: real-time acquisition of a real picture, initialization of a virtual target and preliminary superimposition and fusion of the virtual target with an initial real picture; the last virtual target and real picture superimposition and fusion update result is recorded as a reference picture, the difference between the current real picture and the reference picture is analyzed, and it is determined whether virtual target and current real picture superimposition and fusion update is needed; if so, the local similarity of the local area between the current real picture and the reference picture is analyzed, and the local consistent area between the current real picture and the reference picture is matched; the corner points of the local consistent area in the current real picture and the reference picture are matched to obtain the relative displacement vector of the device; according to the relative displacement vector, the superimposition and fusion update of the virtual target and the current real picture is carried out.

2. The target overlay method for augmented reality according to claim 1, wherein, The difference between the current real picture and the reference picture is analyzed, including: obtaining the first pixel difference value between all real pictures and their corresponding reference pictures to obtain a picture difference value matrix set and a current picture difference value matrix; the average value of the element value of each picture difference value matrix in the picture difference value matrix set is calculated to obtain the maximum difference value matrix average and the average value of the current picture difference value matrix; the position distribution uniformity of the maximum value point in the current picture difference value matrix is analyzed, and the difference between the current real picture and the reference picture is obtained in combination with the maximum difference value matrix average and the average value of the current picture difference value matrix.

3. The target overlay method for augmented reality according to claim 2, wherein, The position distribution uniformity of the maximum value point in the current picture difference value matrix is analyzed, including: obtaining the nearest distance between each maximum value point and its nearest maximum value point in the current picture difference value matrix, and obtaining the farthest distance between each maximum value point and its farthest maximum value point, calculating the difference between the farthest distance and the nearest distance, and combining the number of diagonal line data in the current picture difference value matrix to obtain the position distribution uniformity of the maximum value point in the current picture difference value matrix.

4. The target overlay method for augmented reality according to claim 3, wherein, It is determined whether virtual target and current real picture superimposition and fusion update is needed, including: presetting a difference threshold; determining whether the difference is greater than or equal to the difference threshold; if so, the virtual target and current real picture superimposition and fusion update is carried out.

5. The target overlay method for augmented reality of claim 1, wherein, The local similarity of the local area between the current real picture and the reference picture is analyzed, including: determining the local overlap mode with the most similar area between the current real picture and the reference picture to obtain an overlap area; obtaining the pixel difference minimum value point of the overlap area in the current real picture and the reference picture; respectively, in the current real picture and the reference picture, gradually expanding the area with the pixel difference minimum value point as the center to obtain the corresponding expansion new point set in the current real picture and the reference picture; analyze the second pixel difference and its distribution between the expansion new point set in the current real picture and the reference picture to obtain the consistency degree of the expansion new point set; according to the consistency degree, in combination with the difference, the local similarity of the local area between the current real picture and the reference picture is obtained.

6. The target overlay method for augmented reality according to claim 5, wherein, According to the consistency degree, the difference degree, and the local similarity between the current real picture and the reference picture, the local similarity between the local region of the current real picture and the reference picture is obtained, including: Based on the consistency degree, the consistency similarity between the region corresponding to the pixel difference value minimum point before the current step region expansion and the newly added point set after the current step region expansion is analyzed; Based on the difference degree, the difference degree similarity between the difference degree between the current real picture and the reference picture and the difference degree between the newly added point set after the current step region expansion in the overlapping region is analyzed; Based on the consistency similarity and the difference degree similarity, the expansion region similarity of the pixel difference value minimum point corresponding region in the current real picture and the reference picture after the current step region expansion is obtained; According to the expansion region similarity, the local similarity between the local region of the current real picture and the reference picture is obtained.

7. The target overlay method for augmented reality according to claim 6, wherein, According to the expansion region similarity, the local similarity between the local region of the current real picture and the reference picture is obtained, including: After each step region expansion, the size of the expansion region similarity value corresponding to the region before and after the region expansion is compared; If the expansion region similarity value becomes larger, the current step region expansion is retained, and the next step region expansion is continued; If the expansion region similarity value becomes smaller, the last step region expansion is retained and the expansion is stopped, and the expansion region similarity corresponding to the last step region expansion is recorded as the local similarity between the local region of the current real picture and the reference picture.

8. The target overlay method for augmented reality of claim 1, wherein, Matching the corner points of the local consistent region in the current real picture and the reference picture, the relative displacement vector of the device is obtained, including: Obtaining the corner points of the current real picture and the reference picture; Based on the corner points, using the corner point matching algorithm, the highest matching degree of the corner point pair of the local consistent region in the current real picture and the reference picture is matched, and the vector from the corner point position on the reference picture to the corner point position on the current real picture in the highest matching degree of the corner point pair is recorded as the displacement vector of the device; According to the displacement vector, combined with the local similarity corresponding to the local consistent region, the relative displacement vector of the device is obtained.

9. A target overlay system for augmented reality, characterized by The system comprises a memory and a processor, wherein: The memory is used to store program code; The processor is used to read the program code stored in the memory and execute the method according to any one of claims 1 to 8.

10. The target overlay system for augmented reality of claim 9, wherein, The processor comprises: A picture acquisition and preliminary superposition fusion module is used to acquire a real picture in real time, initialize a virtual target, and preliminarily superimpose and fuse the virtual target with an initial real picture; A current real picture change degree analysis module is used to record the picture obtained after the virtual target and the real picture are superimposed and fused for updating as a reference picture, analyze the difference degree between the current real picture and the reference picture, and determine whether the virtual target and the current real picture need to be superimposed and fused for updating; The superimposition fusion updating module is configured to analyze local similarity of a local area between the current real picture and the reference picture, match a local consistent area between the current real picture and the reference picture, further match a corner point of the local consistent area in the current real picture and the reference picture to obtain a relative displacement vector of the device, and perform superimposition fusion updating of the virtual target and the current real picture according to the relative displacement vector.

Images