Target superposition method and system for augmented reality

Abstract

The invention relates to the technical field of virtual-real fusion image processing, in particular to a target superposition method and system for augmented reality, and the method comprises the steps: firstly simulating to obtain a virtual route target, and superposing and fusing the virtual route target into an initial reality picture for guiding navigation; analyzing the change degree of the current reality picture, and judging whether the virtual target and the current reality picture need to be superposed, fused and updated or not; if yes, analyzing the similarity between the local areas of the current real picture and the reference picture, and preferentially matching the local areas; the relative displacement of the equipment is evaluated according to the matching result of the angular points in each local area, superposition fusion updating of the virtual target and the current real picture is carried out, a more accurate superposition fusion result of the virtual target is obtained, and the immersion and accuracy of a user during use are enhanced; and on the premise of ensuring the accuracy, the AR operation efficiency is effectively improved, the calculation delay is reduced, and the loss of equipment performance is reduced.

Description

Technical Field

[0001] This invention relates to the field of virtual-real fusion image processing technology, specifically to a target overlay method and system for augmented reality. Background Technology

[0002] Augmented reality (AR) is a novel interactive technology that integrates real-world environments with virtual information. Through computer graphics, optoelectronic displays, and multi-sensor technologies, it simulates and overlays virtual information—information that is difficult to perceive directly—onto real-world scenes, allowing users to interact with virtual objects in real-time within a real-world context. This technology not only achieves the organic integration of virtual elements and the physical environment but also, through 3D registration and real-time tracking, convinces users that virtual content is part of their real-world environment, thus providing an immersive experience that transcends reality. Augmented reality requires visual fusion of the virtual and real worlds in three-dimensional space, and it necessitates specialized display devices to meet these fusion requirements. Currently, common augmented reality display devices can be broadly categorized into three types: wearable devices, mobile handheld displays, and spatial augmentation devices.

[0003] In the process of merging virtual targets with real-world scenes, users often move the augmented reality (AR) device's view to observe more virtual information. For example, in an exhibition hall, the virtual projected target may be too large for the device's camera to capture completely, or the user's position in AR navigation may change in real time, causing the AR device's view to change rapidly. The virtual navigation indicators need to be updated and merged to keep up with the user's movements for a smoother user experience. However, when the user moves quickly, the captured view changes significantly, making it difficult to match relative positions within the image and re-merge the virtual view in a timely manner.

[0004] To address the issue of displacement deviations during the overlay and fusion of real-world images captured by augmented reality devices due to rapid changes in the captured images and virtual target information, existing technologies often rely on the identification and matching of corner points before and after changes in the real-world image to assess the relative displacement direction and distance, thereby adjusting the position of the virtual target within the changed real-world image. However, when the shooting perspective changes rapidly, the real-world image exhibits unique corner points in each image due to the change in lens position, leading to inaccurate corner point matching. This results in low precision in subsequent operations and inaccurate overlay and fusion results. Summary of the Invention

[0005] In order to solve the above technical problems, the purpose of this invention is to provide a target overlay method and system for augmented reality.

[0006] According to a first aspect of the present invention, a target overlay method for augmented reality is provided, the specific technical solution of which is as follows: The system acquires real-time images, initializes virtual targets, and initially overlays and blends them with the initial real-world images. The image obtained after the last overlay and fusion update of the virtual target and the real image is recorded as the reference image. The difference between the current real image and the reference image is analyzed to determine whether it is necessary to overlay and fusion update the virtual target and the current real image. If so, analyze the local similarity of local regions between the current real-world image and the reference image, and match the local consistent regions between the current real-world image and the reference image; By matching the corner points of the locally consistent region within the current real-world image and the reference image, the relative displacement vector of the device is obtained; Based on the relative displacement vector, the virtual target and the current real-world scene are superimposed, fused, and updated.

[0007] In some embodiments of the present invention, analyzing the degree of difference between the current real-world image and the reference image includes: Obtain the first pixel difference between all real-world images and their corresponding reference images to obtain the set of image difference matrices and the current image difference matrix; Calculate the mean value of each element of the image difference matrix in the set of image difference matrices to obtain the mean value of the maximum difference matrix and the mean value of the current image difference matrix; By analyzing the uniformity of the distribution of the maximum points in the current image difference matrix, and combining the mean of the maximum difference matrix and the mean of the current image difference matrix, the degree of difference between the current real image and the reference image is obtained.

[0008] In some embodiments of the present invention, analyzing the uniformity of the positional distribution of the maximum points in the current image difference matrix includes: Obtain the nearest distance between each maximum point and its nearest maximum point in the current image difference matrix, and obtain the farthest distance between each maximum point and its farthest maximum point. Calculate the difference between the farthest distance and the nearest distance. Combine this with the number of data points passed through the diagonal of the current image difference matrix to obtain the uniformity of the positional distribution of the maximum points in the current image difference matrix.

[0009] In some embodiments of the present invention, determining whether it is necessary to overlay and merge the virtual target with the current real-world image includes: Preset difference threshold; Determine whether the degree of difference is greater than or equal to the degree of difference threshold; If so, then the virtual target will be overlaid and merged with the current real-world scene for an update.

[0010] In some embodiments of the present invention, analyzing the local similarity of local regions between the current real-world image and the reference image includes: Determine the local overlap pattern that maximizes the similarity between the current real-world image and the reference image to obtain the overlapping area; Obtain the minimum pixel difference points of the overlapping regions in the current real-world image and the reference image, respectively; In both the current real-world image and the reference image, the region is gradually expanded centered on the point where the pixel difference is minimal, to obtain the corresponding set of newly expanded points in the current real-world image and the reference image. By analyzing the second pixel difference and its distribution between the expanded new point set and the current real-world image and the reference image, the consistency of the expanded new point set is obtained. Based on the degree of consistency and the degree of difference, the local similarity of local areas between the current real-world image and the reference image is obtained.

[0011] In some embodiments of the present invention, based on the degree of consistency and combined with the degree of difference, a local similarity of a local region between the current real-world image and the reference image is obtained, including: Based on the aforementioned degree of consistency, the similarity of the degree of consistency between the region corresponding to the minimum pixel difference point before the current step region expansion and the newly added point set in the current step expansion is analyzed. Based on the difference degree, analyze the similarity between the difference degree between the current real scene and the reference scene and the difference degree between the newly added point set in the current step expansion in the overlapping area; By combining the similarity of consistency and the similarity of difference, the similarity of the expanded regions corresponding to the minimum pixel difference point after the current step region expansion is obtained in the current real-world image and the reference image. Based on the expanded region similarity, the local similarity of local regions between the current real-world image and the reference image is obtained.

[0012] In some embodiments of the present invention, obtaining the local similarity of a local region between the current real-world image and the reference image based on the expanded region similarity includes: After each region expansion step, the similarity values ​​of the expanded regions before and after the expansion are compared. If the similarity value of the expanded region increases, the current region expansion is retained, and the next region expansion is continued. If the similarity value of the expanded region decreases, the previous region expansion is retained and the expansion is stopped. The similarity of the expanded region corresponding to the previous region expansion is recorded as the local similarity of the local region between the current real scene and the reference scene.

[0013] In some embodiments of the present invention, matching the corner points of the locally consistent region within the current real-world image and the reference image to obtain the relative displacement vector of the device includes: Obtain the corner points of the current real-world image and the reference image; Based on the corner points, a corner point matching algorithm is used to match the corner point pairs with the highest matching degree in the current real-world image and the reference image. The vector from the corner point position in the reference image to the corner point position in the current real-world image in the corner point pair with the highest matching degree is recorded as the device displacement vector. Based on the displacement vector and the local similarity corresponding to the locally consistent region, the relative displacement vector of the device is obtained.

[0014] According to a second aspect of the present invention, a target overlay system for augmented reality is provided, comprising: a memory and a processor, wherein: The memory is used to store program code; The processor is configured to read program code stored in the memory and execute the method described in the first aspect of the present invention.

[0015] In some embodiments of the present invention, the processor includes: The image acquisition and preliminary overlay and fusion module is used to acquire real-time images, initialize virtual targets, and preliminarily overlay and fuse them with the initial real-world images; The current reality scene change analysis module is used to record the image obtained after the last virtual target and reality scene were superimposed and merged as a reference image, analyze the difference between the current reality scene and the reference image, and determine whether the virtual target and current reality scene need to be superimposed and merged. The overlay and fusion update module is used to analyze the local similarity of local regions between the current real-world image and the reference image, match the local consistent regions between the current real-world image and the reference image, and then match the corner points of the local consistent regions in the current real-world image and the reference image to obtain the relative displacement vector of the device; and perform overlay and fusion update of the virtual target and the current real-world image based on the relative displacement vector.

[0016] Compared with existing technologies, the target overlay method and system for augmented reality provided by this invention have the following beneficial effects: This invention first acquires the user's initial and target positions, simulates a virtual route target, and overlays it onto the user's initial captured real-world image for navigation guidance. As the user's device camera moves, the captured real-world image changes. The degree of change in the real-world image is assessed to determine whether an update to overlay the virtual target onto the current real-world image is necessary. For minor changes, no update is performed, effectively improving the efficiency of the AR algorithm, reducing computational latency, and minimizing device performance degradation while maintaining accuracy. For real-world images requiring overlay updates, the local similarity between the current and reference images of the same local area is used to prioritize matching local areas. The relative device displacement is then assessed based on the matching results of corner points within each local area, allowing for rapid changes in the virtual target's position and overlay updates. This results in more accurate virtual target overlay results, enhancing user immersion and accuracy, and improving the user experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the basic process of a target overlay method for augmented reality provided in an embodiment of the present invention; Figure 2 An example diagram provided by an embodiment of the present invention shows that the displacement of the same real local area and the virtual path relative to the device is substantially consistent between the current real-world image and the reference image; Figure 3 An example diagram illustrating how an overlapping area is formed by relative movement of diagonal points between a current real-world image and a reference image, as provided in an embodiment of the present invention; Figure 4 An example diagram of the displacement vector of a device obtained based on the corner point with the highest matching degree, provided as an embodiment of the present invention; Figure 5 This is a schematic diagram of the basic components of a target overlay system for augmented reality provided in one embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a target overlay method and system for augmented reality proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms such as “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a circuit structure, article, or device comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the article or device that includes that element. Relational terms such as “first” and “second” are used merely to distinguish one entity or operation from another and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0021] The present invention addresses the following scenario: In augmented reality applications such as navigation in libraries or shopping malls, where users need to move frequently and the captured images of the real world change frequently, the relative positions of virtual targets and real-world images are prone to deviation, with large displacement gaps, resulting in a certain delay in updates and affecting the user experience.

[0022] Therefore, the main objective of this invention is to solve the problem that the camera's field of view changes rapidly during the user's rapid movement while using the device, resulting in large and rapid changes in the captured real-world image, which leads to deviations in the relative position when the real-world image is superimposed and merged with the virtual target. The invention aims to accurately obtain a fusion and superposition result in which the relative position of the virtual target changes correctly as the real-world image changes.

[0023] To achieve the above-mentioned objectives, this invention, based on existing technology, further compares the real-world images captured before and after the device moves, and obtains the corresponding positions of the same real-world image in the two images before and after the device moves by subtracting the results of different local areas. This allows for more accurate matching of corner points within local areas, obtaining the relative displacement degree before and after the move, and enabling more accurate virtual target overlay and fusion.

[0024] The following description, in conjunction with the accompanying drawings, details a specific scheme of the target overlay method for augmented reality provided by the present invention.

[0025] Please see Figure 1 This illustrates the basic flow of a target overlay method for augmented reality provided by an embodiment of the present invention.

[0026] like Figure 1 As shown, an embodiment of the present invention provides a target overlay method for augmented reality, specifically including: S100: Acquires real-time images, initializes virtual targets, and performs preliminary overlay and fusion with the initial real-world images.

[0027] The system acquires real-time images of the real world, initializes virtual targets, and initially overlays and blends them with the initial real-world imagery. Specifically, it first captures an initial real-world image of the user's initial location using the device's camera; then, based on the user's initial location, the initial real-world image captured by the device's camera, and the set target location, it uses AR (Augmented Reality Navigation) technology and the 3D space of the user's initial scene to model the surrounding environment, acquiring images of the user's surroundings captured by the device's camera; and finally, it uses an AR algorithm (Autoregressive algorithm) to initialize virtual targets and accurately overlay virtual navigation markers onto the initial real-world imagery to guide the user towards their set target location. During the user's movement, it continuously acquires real-world images captured by the user's device's camera.

[0028] At this point, the initial virtual target overlay and fusion results, as well as the real-time updated real-world image, were obtained.

[0029] Because AR algorithms require a large amount of complex calculations, they are demanding in terms of time and computing power. Real-time generation and overlay fusion of virtual targets using AR algorithms based on real-world images is time-consuming and detrimental to device performance. During actual user movement, small displacements are common, and the same real-world area exists before and after the movement. This allows for the determination of the device's relative displacement, enabling rapid adjustment and overlay fusion of virtual target positions. Therefore, during the AR algorithm's adjustment intervals, the virtual target overlay fusion can be updated based on the existing overlay results and changes in the real-world image, reducing performance loss and computational latency while maintaining accuracy. In the embodiments of this invention, steps S200 to S500 are specifically included.

[0030] S200: Take the image obtained after the last overlay and fusion update of the virtual target and the real image as the reference image, analyze the difference between the current real image and the reference image, and determine whether it is necessary to overlay and fusion update the virtual target and the current real image.

[0031] First, the image obtained after the last virtual target and real-world image overlay and fusion update operation is recorded as the reference image. The current real-world image is recorded in real time and recorded as the current real-world image. When the user moves or rotates the AR device, the camera on the device moves accordingly, and the real-world image changes accordingly. Therefore, there is a high probability that there will be a significant difference between the two real-world images before and after the movement. Therefore, by analyzing the degree of difference between the current real-world image and the reference image, it can be determined whether a virtual target and current real-world image overlay and fusion update operation is necessary. In other words, it can be assessed whether the current real-world image has changed significantly to the point that a virtual target and current real-world image overlay and fusion update operation is required.

[0032] Furthermore, the analysis examines the degree of difference between the current real-world image and the reference image, including: First, obtain the first pixel difference between all real-world images and their corresponding reference images, resulting in a set of image difference matrices and the current image difference matrix. Specifically, at regular intervals (user-defined, this example uses once per second), record the current real-world image and calculate the first pixel difference between the real-world image and the reference image at that moment using the frame difference method (the obtained pixel differences are all absolute values, which will not be elaborated further below), to obtain the image difference matrix; assuming that a total of [number] real-world images have been captured up to the current time. The set of image difference matrices corresponding to each real-world image up to the current real-world image is denoted as [missing information]. ,in, Represents the set of image difference matrices. Indicates the first The image difference matrix corresponding to the actual image; This represents the total number of real-world images captured up to the current scene. The current scene difference matrix is ​​then... .

[0033] Then, the value of each element in the image difference matrix represents the difference in the same area between the two images; a larger value indicates a greater pixel difference at that point between the two images. Therefore, in some embodiments of the present invention, the mean value of the elements of each image difference matrix in the set of image difference matrices is calculated (denoted as ). ), to obtain the mean of the maximum difference matrix (denoted as ). ), and obtain the mean value of the elements of the current frame difference matrix (denoted as ), ).

[0034] Furthermore, the maxima in the image difference matrix represent the points of greatest difference within a local region. In two images with significant differences, each local region exhibits substantial differences, and the maxima are likely numerous and evenly distributed. Therefore, in some embodiments of this invention, the evenness of the distribution of maxima in the current image difference matrix is ​​analyzed. Specifically, the current image difference matrix is ​​taken as... The number of mesomaxima is denoted as . Get the current frame difference matrix. The nearest distance between each local maximum and its nearest local maximum (denoted as ) ), and obtain the farthest distance between each maximum point and its farthest maximum point (denoted as ), ), calculate the difference between the farthest distance and the nearest distance, denoted as . ,in, Represents the current frame difference matrix Inner The difference between the farthest and nearest distances corresponding to each maximum point. Represents the current frame difference matrix Inner The farthest distance corresponding to each maximum point Represents the current frame difference matrix Inner The nearest distance to each maximum point; obtain the current frame difference matrix. The diagonal line passes through the number of data points. Combined with the first The difference between the farthest and nearest distances corresponding to each maximum point Difference matrix with current image The diagonal line passes through half the number of data points. Iterate through all the maximum points in the current frame difference matrix to obtain the current frame difference matrix. Uniformity of the location distribution of the maximum points.

[0035] Finally, by combining the uniformity of the location distribution of the maximum points in the current image difference matrix, the mean of the maximum difference matrix, and the mean of the current image difference matrix, the degree of difference between the current real-world image and the reference image is obtained. Then, the current real-world image is constructed. The formula for calculating the difference between the image and the reference image is: ; In the formula, Indicates the degree of difference between the current real-world image and the reference image; This represents the mean value of the elements in the current frame difference matrix; This represents the mean of the maximum difference matrix; Represents the current reality scene Inner The difference between the farthest and nearest distances corresponding to each maximum point; Represents the current frame difference matrix The number of data points that the diagonal line passes through; Represents the current reality scene Number of internal maxima; Represents a linear normalization function; This indicates taking the absolute value.

[0036] and Each represents the degree of difference between the corresponding real-world image and the reference image, and the ratio between the two. Larger and close to This indicates that the current real-world image differs more significantly from the reference image compared to all real-world images obtained within that time period. The larger the value, the farther away the local maxima are in the image. This means that the more discrete the distribution of the maxima in the image, the better. The sum of the dispersion of all the maximum points is used to represent the uniformity of the location distribution of the maximum points in the current image difference matrix. The larger this value is, the more maximum points there are in the current real image and the more dispersed their distribution is, which means that the difference between the current real image and the reference image is greater.

[0037] Obtain the degree of difference between the current real-world image and the reference image. , The larger the value, the greater the difference between the current real-world image and the reference image, and the more necessary it is to perform a virtual target overlay and fusion update operation. Therefore, after obtaining the difference degree between the current real-world image and the reference image, it is determined whether a virtual target overlay and fusion update is needed based on the difference degree. Specifically, a preset difference degree threshold is set, which can be 0.65; it is then checked whether the difference degree is greater than or equal to the difference degree threshold; if so, i.e. If the current reality is significantly different from the reference image, a virtual target overlay and fusion update operation is needed; otherwise, i.e. They believe that the current real-world scene is not significantly different from the reference scene, and therefore there is no need to perform an overlay and fusion update operation between the virtual target and the current real-world scene.

[0038] S300: Analyze the local similarity between local areas of the current real-world image and the reference image, and match the local consistent areas between the current real-world image and the reference image.

[0039] According to step S200 above, if it is determined that the current real-world image differs significantly from the reference image, a superposition and fusion update operation between the virtual target and the current real-world image is required. Although there are significant differences between the current real-world image and the reference image, since they are two images obtained by camera displacement within the same real-world scene, there is a high probability that several local areas of the same real-world image are located in different positions in the two images, such as... Figure 2As shown, the relative change in its position can represent the change of the virtual target relative to the camera. Therefore, it is necessary to first identify and match the local area of ​​the same real scene in the current real scene and the reference scene.

[0040] Based on the above analysis, in the embodiments of the present invention, when performing the overlay and fusion update operation of the virtual target and the current real-world image, the local similarity of local regions between the current real-world image and the reference image is first analyzed, and local consistent regions between the current real-world image and the reference image are matched. Further, this includes: First, determine the local overlap pattern that maximizes the similarity between the current scene and the reference scene, thus identifying the overlapping areas. Specifically, due to the displacement of the shooting device, there are consistent areas and unique areas between the current scene and the reference scene. Therefore, first, roughly determine the local overlap pattern that maximizes the consistent areas between the current scene and the reference scene. Move relative to each other from opposite corners of the current scene and the reference scene, such as... Figure 3 Starting from the bottom right point of image a (reference image) and the top left point of image b (current reality image), image b is successively shifted upwards and to the left by one unit distance to obtain the overlapping area of ​​image a and image b each time they are shifted.

[0041] Furthermore, the minimum pixel difference points of the overlapping region within the current real-world image and the reference image are obtained. Specifically, the number of minimum pixel difference points and the average minimum pixel difference points of the overlapping region between the current real-world image and the reference image are obtained. The overlapping method corresponding to the minimum average minimum pixel difference point is selected as the local overlapping method with the most similar regions. When the average minimum pixel difference point is the same, the overlapping method corresponding to the maximum number of minimum pixel difference points is selected as the local overlapping method with the most similar regions. The overlapping region corresponding to the local overlapping method is obtained, and the minimum pixel difference points of the overlapping region within the current real-world image and the reference image are obtained.

[0042] Then, in both the current real-world image and the reference image, the region is gradually expanded centered on the point with the minimum pixel difference, resulting in a set of newly added points within both the current real-world image and the reference image. Specifically, based on the minimum pixel difference points within the overlapping region selected above, the same real-world region containing each minimum pixel difference point is expanded and divided. Taking a specific minimum pixel difference point as an example, the neighborhood range is gradually expanded outward from the corresponding points in both the current real-world image and the reference image, resulting in a set of newly added points within both the current real-world image and the reference image.

[0043] Furthermore, the smaller the pixel difference between different locations in the current real-world image and the reference image, the higher the similarity at that point, and the more likely they are points from the same local real-world region. Similarly, the smaller the pixel difference around the point with the minimum pixel difference and the greater the clustering of these points, the higher the similarity between the current real-world image and the reference image in that region, and the larger the area of ​​that region, and the more likely it is that the region is from the same local real-world region. Therefore, in some embodiments of the present invention, the consistency of the expanded new point set is obtained by analyzing the second pixel difference and its distribution between the current real-world image and the reference image. Specifically, taking the current step region expansion as an example, the number of expanded new points within the expanded new point set obtained from the current step region expansion is obtained. And obtain the second pixel difference (denoted as ) between the current real-world image and the reference image for all expanded new points in the expanded new point set. The total pixel difference of all newly added points within the expanded point set is calculated and denoted as . Further, obtain the maximum and minimum values ​​of the second pixel difference corresponding to the newly added points in the current step of expansion, and calculate the distance between the newly added points corresponding to the maximum and minimum values ​​of the second pixel difference (denoted as ). ), and the difference between the maximum and minimum values ​​of the second pixel difference (denoted as ). Therefore, the formula for calculating the consistency of the newly added point set in the expansion is as follows: ; In the formula, Indicates the degree of consistency of the newly added point set during expansion; This represents the total pixel difference among all newly added points within the expanded point set. This represents the distance between the newly added points corresponding to the maximum and minimum values ​​of the second pixel difference in the set of expanded new points. This represents the difference between the maximum and minimum values ​​of the second pixel difference in the newly added point set.

[0044] The total pixel difference of the expanded new point set is used to visually represent the size of the pixel difference in the expanded new point set. The smaller the value, the lower the overall pixel difference of the expanded new point set and the higher the similarity. and The ratio represents the distance between the points with the largest and smallest pixel differences in the newly added points set, and the magnitude of their difference. A larger ratio indicates a greater distance between the two newly added points and a smaller difference between them, suggesting a more uniform and consistent set of differences. In summary, the consistency of the newly added point set... The larger the value, the lower the overall pixel difference value of the expanded new point set, and the higher the consistency, indicating that the similarity of the regions where they are located is higher.

[0045] Using the same method, calculate the consistency of the original point set (the point set formed by the regions corresponding to the minimum pixel differences before the current step region expansion), denoted as . .

[0046] Finally, based on the degree of consistency and combined with the degree of difference, the local similarity of local areas between the current real-world image and the reference image is obtained. Furthermore, this includes: First, based on the degree of consistency, we analyze the similarity between the regions corresponding to the minimum pixel difference points before the current step of region expansion and the newly added point set in the current step, that is, the degree of consistency of the point sets formed by the regions corresponding to the minimum pixel difference points before the current step of region expansion. The degree of consistency between the current step expansion and the newly added point set. The degree of similarity between them.

[0047] Simultaneously, based on the difference degree, the similarity between the difference degree between the current real image and the reference image and the difference degree between the newly added point set in the current step of expansion in the overlapping area is analyzed. Specifically, the difference degree between the local areas formed after the current area expansion is completed by the pixel difference minimum point in the current real image and the reference image, respectively, is calculated using the same method as in step S200, and is denoted as... .

[0048] Then, combining the similarity of consistency and the similarity of difference, the similarity of the expanded regions corresponding to the minimum pixel difference points after the current step expansion is obtained in the current real-world image and the reference image. Therefore, the formula for calculating the similarity of the expanded regions corresponding to the minimum pixel difference points after the current step expansion in the current real-world image and the reference image is as follows: ; In the formula, This indicates the similarity between the region corresponding to the minimum pixel difference point after the current step region expansion and the expanded region in the current real-world image and the reference image. Indicates the degree of difference between the current real-world image and the reference image; It represents the degree of difference between the local areas formed in the current real-world image and the reference image after the pixel difference minimum point completes the current region expansion; Indicates the degree of consistency of the newly added point set during expansion; This indicates the degree of consistency of the point set formed by the region corresponding to the minimum pixel difference point before the current step of region expansion; This indicates taking the absolute value.

[0049] The larger the value, the better. Compared to The smaller the value, the smaller the difference between the local area obtained by the minimum pixel difference point after the current step of region expansion and the overall image. The higher the similarity between the current real image and the corresponding local area in the reference image. The larger the value, the higher the similarity between the current real-world image and the corresponding expanded point set region in the reference image; the larger the difference between the two. The smaller the value, the more similar the difference between the newly expanded point set region and the original region before expansion. The larger the expanded region is and the lower the difference within the region, the higher the overall similarity of the local region.

[0050] Furthermore, based on the similarity of the expanded regions, the local similarity of local regions between the current real-world image and the reference image is obtained. Specifically, after each step of region expansion, the similarity values ​​of the expanded regions before and after expansion are compared; if the similarity value of the expanded regions increases, the current region expansion is retained, and the next step of region expansion continues (expanding outwards to a surrounding neighborhood); if the similarity value of the expanded regions decreases, the previous region expansion is retained and expansion stops, and the similarity value of the expanded regions corresponding to the previous step is recorded as the local similarity of local regions between the current real-world image and the reference image.

[0051] Similarly, for all the minimum pixel difference points in the overlapping area obtained from the current real image and the reference image, the region is expanded. After the region expansion is completed, the final expanded region corresponding to each minimum pixel difference point is obtained, which is recorded as the local consistent region between the current real image and the reference image, and the local similarity corresponding to each local consistent region is obtained.

[0052] S400: Match the corner points of the local consistent region within the current real-world image and the reference image to obtain the relative displacement vector of the device.

[0053] By matching the corner points of locally consistent regions within the current real-world image and the reference image, the relative displacement vector of the device is obtained. Further, this includes: First, obtain the corner points of the current real-world image and the reference image. Specifically, use a corner detection algorithm to identify and mark the corner positions of the current real-world image and the reference image.

[0054] Then, based on corner points, a corner point matching algorithm is used to match the corner point pairs with the highest matching degree in the current real-world image and the reference image within a locally consistent region. The vector from the top corner point in the reference image (the corner point corresponding to the highest matching degree in the reference image) to the top corner point in the current real-world image (the corner point corresponding to the highest matching degree in the current real-world image) in the highest matching degree corner point pair is recorded as the device's displacement vector. Figure 4 As shown.

[0055] Finally, based on the displacement vector and the local similarity corresponding to the locally consistent regions, the relative displacement vector of the device is obtained. Specifically, it is assumed that a total of [number] displacement vectors are obtained. There are several locally consistent regions, and the local similarity of each locally consistent region is denoted as . The sum of local similarities is denoted as Combining the displacement vector and the sum of local similarities, the relative displacement vector of the device is obtained as follows: ; In the formula, Represents the relative displacement vector of the device; Indicates the first Local similarity of locally consistent regions; Indicates all The sum of local similarities of locally consistent regions; This represents the device's displacement vector (the vector from the reference top corner of the screen to the current top corner of the real screen). Represents the current reality scene The number of locally consistent areas between the reference image and the reference image.

[0056] By the relative value of local similarity By weighting the displacement vectors of the equipment, we obtain the relative displacement vector of the equipment. The larger this value is, the more obvious the movement of the equipment.

[0057] S500: Based on the relative displacement vector, the virtual target is superimposed and merged with the current real-world scene for updating.

[0058] Based on the relative displacement vector, the virtual target and the current real-world image are superimposed and merged. This means that the relative position of the virtual target in the current real-world image is shifted according to the relative displacement vector before the superimposition and fusion update operation is performed to obtain the updated superimposed and fused effect, which guides the user to move towards the target position.

[0059] Based on the same inventive concept as the above method, this embodiment also provides a target overlay system for augmented reality.

[0060] Please see Figure 5 This illustrates the basic components of a target overlay system for augmented reality provided by an embodiment of the present invention.

[0061] like Figure 5 As shown, a target overlay system for augmented reality includes: a memory 10 and a processor 20, wherein: Memory 10 is used to store program code; The processor 20 is used to read the program code stored in the memory 10 and execute it to acquire the real-time image, initialize the virtual target and perform a preliminary overlay and fusion with the initial real-time image; record the image obtained after the last overlay and fusion update of the virtual target and the real-time image as the reference image, analyze the difference between the current real-time image and the reference image, and determine whether it is necessary to perform an overlay and fusion update of the virtual target and the current real-time image; if so, analyze the local similarity of local areas between the current real-time image and the reference image, and match the local consistent areas between the current real-time image and the reference image; match the corner points of the local consistent areas in the current real-time image and the reference image to obtain the relative displacement vector of the device; and perform the overlay and fusion update of the virtual target and the current real-time image based on the relative displacement vector.

[0062] Furthermore, the processor 20 includes an image acquisition and preliminary overlay fusion module 21, a current real-world image change analysis module 22, and an overlay fusion update module 23, wherein: The image acquisition and preliminary overlay and fusion module 21 is used to acquire real-time images, initialize virtual targets, and preliminarily overlay and fuse them with the initial real-world images; The current reality scene change analysis module 22 is used to record the image obtained after the last virtual target and reality scene overlay and fusion update as the reference image, analyze the difference between the current reality scene and the reference image, and determine whether the virtual target and current reality scene overlay and fusion update is needed. The overlay and fusion update module 23 is used to analyze the local similarity of local areas between the current real scene and the reference scene, match the local consistent areas between the current real scene and the reference scene; then match the corner points of the local consistent areas in the current real scene and the reference scene to obtain the relative displacement vector of the device; and perform overlay and fusion update of the virtual target and the current real scene based on the relative displacement vector.

[0063] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0064] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A target overlay method for augmented reality, characterized in that, The method includes: The system acquires real-time images, initializes virtual targets, and initially overlays and blends them with the initial real-world images. The image obtained after the last overlay and fusion update of the virtual target and the real image is recorded as the reference image. The difference between the current real image and the reference image is analyzed to determine whether it is necessary to overlay and fusion update the virtual target and the current real image. If so, analyze the local similarity of local regions between the current real-world image and the reference image, and match the local consistent regions between the current real-world image and the reference image; By matching the corner points of the locally consistent region within the current real-world image and the reference image, the relative displacement vector of the device is obtained; Based on the relative displacement vector, the virtual target and the current real-world scene are superimposed, fused, and updated.

2. The target overlay method for augmented reality according to claim 1, characterized in that, Analyze the differences between the current real-world scene and the reference scene, including: Obtain the first pixel difference between all real-world images and their corresponding reference images to obtain the set of image difference matrices and the current image difference matrix; Calculate the mean value of each element of the image difference matrix in the set of image difference matrices to obtain the mean value of the maximum difference matrix and the mean value of the current image difference matrix; By analyzing the uniformity of the distribution of the maximum points in the current image difference matrix, and combining the mean of the maximum difference matrix and the mean of the current image difference matrix, the degree of difference between the current real image and the reference image is obtained.

3. The target overlay method for augmented reality according to claim 2, characterized in that, The analysis includes assessing the uniformity of the distribution of maximum points in the current image difference matrix, including: Obtain the nearest distance between each maximum point and its nearest maximum point in the current image difference matrix, and obtain the farthest distance between each maximum point and its farthest maximum point. Calculate the difference between the farthest distance and the nearest distance. Combine this with the number of data points passed through the diagonal of the current image difference matrix to obtain the uniformity of the positional distribution of the maximum points in the current image difference matrix.

4. The target overlay method for augmented reality according to claim 3, characterized in that, Determining whether a virtual target needs to be overlaid and merged with the current real-world image includes: Preset difference threshold; Determine whether the degree of difference is greater than or equal to the degree of difference threshold; If so, then the virtual target will be overlaid and merged with the current real-world scene for an update.

5. The target overlay method for augmented reality according to claim 1, characterized in that, Analyzing the local similarity of local regions between the current real-world image and the reference image includes: Determine the local overlap pattern that maximizes the similarity between the current real-world image and the reference image to obtain the overlapping area; Obtain the minimum pixel difference points of the overlapping regions in the current real-world image and the reference image, respectively; In both the current real-world image and the reference image, the region is gradually expanded centered on the point where the pixel difference is minimal, to obtain the corresponding set of newly expanded points in the current real-world image and the reference image. By analyzing the second pixel difference and its distribution between the expanded new point set and the current real-world image and the reference image, the consistency of the expanded new point set is obtained. Based on the degree of consistency and the degree of difference, the local similarity of local areas between the current real-world image and the reference image is obtained.

6. The target overlay method for augmented reality according to claim 5, characterized in that, Based on the degree of consistency and the degree of difference, the local similarity of local regions between the current real-world image and the reference image is obtained, including: Based on the aforementioned degree of consistency, the similarity of the degree of consistency between the region corresponding to the minimum pixel difference point before the current step region expansion and the newly added point set in the current step expansion is analyzed. Based on the difference degree, analyze the similarity between the difference degree between the current real scene and the reference scene and the difference degree between the newly added point set in the current step expansion in the overlapping area; By combining the similarity of consistency and the similarity of difference, the similarity of the expanded regions corresponding to the minimum pixel difference point after the current step region expansion is obtained in the current real-world image and the reference image. Based on the expanded region similarity, the local similarity of local regions between the current real-world image and the reference image is obtained.

7. The target overlay method for augmented reality according to claim 6, characterized in that, Based on the expanded region similarity, the local similarity of local regions between the current real-world image and the reference image is obtained, including: After each region expansion step, the similarity values ​​of the expanded regions before and after the expansion are compared. If the similarity value of the expanded region increases, the current region expansion is retained, and the next region expansion is continued. If the similarity value of the expanded region decreases, the previous region expansion is retained and the expansion is stopped. The similarity of the expanded region corresponding to the previous region expansion is recorded as the local similarity of the local region between the current real scene and the reference scene.

8. The target overlay method for augmented reality according to claim 1, characterized in that, Matching the corner points of the locally consistent region within the current real-world image and the reference image to obtain the relative displacement vector of the device includes: Obtain the corner points of the current real-world image and the reference image; Based on the corner points, a corner point matching algorithm is used to match the corner point pairs with the highest matching degree in the current real-world image and the reference image. The vector from the corner point position in the reference image to the corner point position in the current real-world image in the corner point pair with the highest matching degree is recorded as the device displacement vector. Based on the displacement vector and the local similarity corresponding to the locally consistent region, the relative displacement vector of the device is obtained.

9. A target overlay system for augmented reality, characterized in that, The system includes: a memory and a processor, wherein: The memory is used to store program code; The processor is configured to read program code stored in the memory and execute the method as described in any one of claims 1 to 8.

10. The target overlay system for augmented reality according to claim 9, characterized in that, The processor includes: The image acquisition and preliminary overlay and fusion module is used to acquire real-time images, initialize virtual targets, and preliminarily overlay and fuse them with the initial real-world images; The current reality scene change analysis module is used to record the image obtained after the last virtual target and reality scene were superimposed and merged as a reference image, analyze the difference between the current reality scene and the reference image, and determine whether the virtual target and current reality scene need to be superimposed and merged. The overlay and fusion update module is used to analyze the local similarity of local regions between the current real-world image and the reference image, match the local consistent regions between the current real-world image and the reference image, and then match the corner points of the local consistent regions in the current real-world image and the reference image to obtain the relative displacement vector of the device; and perform overlay and fusion update of the virtual target and the current real-world image based on the relative displacement vector.

Images