Display control method and device based on augmented reality equipment

Abstract

The embodiment of the invention discloses a display control method and device based on augmented reality equipment. According to the technical scheme provided by the embodiment of the invention, the first image obtained by carrying out image shooting on the text plane by the camera is obtained, the second image is converted according to the preset homography matrix and the target image size to obtain the tracking target image, and feature point matching is carried out on the tracking target image and the first image; the method comprises the steps of obtaining a target transformation matrix, determining a target tracking position corresponding to a tracking target image in a camera coordinate system according to the target transformation matrix, and displaying a preset image on a screen according to the target tracking position, so that information related to a text recognition result can be correctly superposed on a real text object for display. And the use experience of the user wearing the augmented reality equipment is further improved.

Description

Technical Field

[0001] This application relates to augmented reality technology, and more particularly to a display control method and apparatus based on an augmented reality device. Background Technology

[0002] Augmented Reality (AR) technology is a technique that cleverly blends virtual information with the real world. With the development of AR technology, various AR devices have appeared in people's daily lives, such as AR glasses and AR headsets. The goal of AR technology is to overlay the virtual world onto the real world on the screen of an AR device and enable interaction.

[0003] Optical Character Recognition (OCR) technology utilizes image processing, pattern recognition, and machine learning to identify features such as the shape, size, arrangement, and color of characters, converting text in an image into character codes or text data for searching, editing, storage, analysis, and sharing. Currently, augmented reality (AR) devices generally rely on traditional OCR systems for text recognition and related information display. For example, AR devices use cameras to capture images, perform text recognition, and then directly display the results on the screen. The displayed results are spatially separated from the actual text objects, resulting in poor display quality and a negative user experience. Summary of the Invention

[0004] This application provides a display control method and apparatus based on an augmented reality device to solve the technical problem in the related art where the information related to text recognition results displayed in the augmented reality device is separated from the real text object in space, which affects the user experience. The method enables the information related to text recognition results to be correctly superimposed on the real text object for display, thereby improving the user experience when wearing the augmented reality device.

[0005] In a first aspect, embodiments of this application provide a display control method based on an augmented reality device, the augmented reality device being equipped with a camera and a screen, the display control method comprising:

[0006] Acquire a first image obtained by the camera capturing an image of the text plane;

[0007] Based on the preset homography matrix and the target image size, the second image is transformed to obtain the tracking target image. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is the first image or an image captured by the augmented reality device before the first image and subjected to text recognition. The homography matrix records the mapping relationship used to transform the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system.

[0008] Feature point matching is performed on the tracking target image and the first image to obtain a target transformation matrix. The target transformation matrix records the mapping relationship used to transform the tracking target image from the text plane coordinate system to the camera coordinate system.

[0009] The target tracking position of the target image in the camera coordinate system is determined based on the target transformation matrix.

[0010] A preset image is displayed on the screen based on the target tracking position.

[0011] This application embodiment acquires a first image obtained by a camera capturing an image of a text plane. A second image is then transformed according to a preset homography matrix and the target image size to obtain a tracking target image. Feature point matching is performed between the tracking target image and the first image to obtain a target transformation matrix. The target tracking position of the tracking target image in the camera coordinate system is determined based on the target transformation matrix, and a preset image is displayed on the screen according to the target tracking position. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition. The homography matrix can be used to transform the text recognition result... The corresponding text image is transformed from the camera coordinate system to the text plane coordinate system, so that the tracking target image can correctly reflect the result of transforming the text recognition result of the second image to the text plane coordinate system. The target transformation matrix obtained by matching the feature points of the tracking target image and the first image can accurately transform the tracking target image to the camera coordinate system, and can accurately determine the target tracking position corresponding to the tracking target image in the camera coordinate system. When the user wears the augmented reality device, the image displayed on the screen observed by the user can be correctly superimposed on the real text object corresponding to the text recognition result, so that the information related to the text recognition result can be correctly superimposed on the real text object for display, further improving the user experience of wearing the augmented reality device.

[0012] Furthermore, the target image size includes the target image width and the target image height. Before converting the second image to obtain the tracking target image based on the preset homography matrix and the target image size, the method further includes:

[0013] The width of the target image is determined based on the first side length of the image text positioning box, and the image text positioning box is determined based on the text recognition result of the second image;

[0014] The height of the target image is determined by multiplying the ratio of the height to the width of the tracking target positioning box with the width of the target image. The tracking target positioning box is obtained by transforming the image text positioning box to the text plane coordinate system.

[0015] As described above, by determining the target image width using the first side length of the image text positioning box, and by determining the target image height based on the height and width of the tracking target positioning box and the target image width, the target image size can be accurately determined, thereby improving the accuracy of acquiring and tracking target images.

[0016] Furthermore, before converting the second image according to the preset homography matrix and the target image size to obtain the tracking target image, the method further includes:

[0017] The homography matrix is ​​determined based on the tracking target positioning box corresponding to the text recognition result of the second image in the text plane coordinate system, the intrinsic parameter matrix of the camera, and the preset unit conversion ratio between image distance units and spatial distance units. The image distance units are used to represent distance units on the image, and the spatial distance units are used to represent distance units in space.

[0018] As described above, the homography matrix can be accurately determined by tracking the target positioning box, the intrinsic parameter matrix of the camera, and the unit conversion ratio, thereby improving the accuracy of acquiring the tracked target image.

[0019] Furthermore, determining the target tracking position of the target image in the camera coordinate system based on the target transformation matrix includes:

[0020] Based on the target transformation matrix, the text recognition result of the second image is transformed into the tracking target location box corresponding to the text plane coordinate system to the camera coordinate system, so as to obtain the target tracking position of the tracking target image corresponding to the camera coordinate system.

[0021] As described above, by transforming the target positioning box to the camera coordinate system using the target transformation matrix, the target tracking position can be accurately obtained, which can effectively improve the accuracy of acquiring the target image.

[0022] Furthermore, displaying a preset image on the screen according to the target tracking position includes:

[0023] Based on the intrinsic and extrinsic parameter matrices of the rendering camera, the target tracking position is transformed from the camera coordinate system to the rendering camera coordinate system to obtain the rendering position information;

[0024] A preset image is displayed on the screen based on the rendering position information.

[0025] As described above, by transforming the target tracking position from the camera coordinate system to the rendering camera coordinate system using the intrinsic and extrinsic parameter matrices of the rendering camera, the rendering position information is obtained. This allows the preset image to be correctly superimposed onto the corresponding position of the text in reality, which can improve the efficiency of positioning switching between virtual and reality, as well as the accuracy of the fused display between virtual and reality.

[0026] Furthermore, after acquiring the first image obtained by the camera capturing the text plane, the method further includes:

[0027] If the text recognition triggering conditions are met, text recognition is performed on the first image to obtain the text recognition result;

[0028] Determine the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system.

[0029] As described above, by performing text recognition on the first image when the text recognition trigger condition is met, the text recognition result is obtained, and the image text positioning box corresponding to the text recognition result is transformed into the text plane coordinate system to obtain the tracking target positioning box. Subsequent images can be tracked based on this tracking target positioning box, without needing to perform text recognition on each first image. This reduces the frequency of text recognition after each acquisition of the first image, reduces resource consumption, and further improves the efficiency of positioning switching between virtual and reality while ensuring the accuracy of the fused display between virtual and reality.

[0030] Furthermore, determining the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system includes:

[0031] Determine the first normal vector of the text plane corresponding to the text recognition result and the first distance from the text plane to the camera;

[0032] Based on the image text localization box corresponding to the text recognition result, the intrinsic parameter matrix of the camera, the first normal vector, and the first distance, the recognition text localization box corresponding to the image text localization box in the camera coordinate system is determined;

[0033] The text plane coordinate system is determined based on the identified text positioning box;

[0034] Determine the tracking target positioning box corresponding to the identified text positioning box in the text plane coordinate system.

[0035] As described above, by using the first normal vector of the text plane corresponding to the text recognition result, the first distance from the text plane to the camera, and the intrinsic parameter matrix of the camera, the recognition text positioning box corresponding to the image text positioning box in the camera coordinate system is determined, and the text plane coordinate system of each recognition text positioning box is determined, as well as the tracking target positioning box corresponding to the recognition text positioning box in the text plane coordinate system is determined. The shape of the obtained tracking target positioning box is closer to the shape of the actual text range on the first image, and the target image can be tracked more accurately on the first image based on the tracking target positioning box, thereby improving the positioning accuracy of the fusion display between virtual and reality.

[0036] Further, determining the corresponding recognized text location box in the camera coordinate system based on the image text location box corresponding to the text recognition result, the intrinsic parameter matrix of the camera, the first normal vector, and the first distance includes:

[0037] Based on the vertex coordinates of the image text location box corresponding to the text recognition result and the intrinsic parameter matrix of the camera, determine the unit direction vector from the origin of the camera coordinate system to the recognized text location box.

[0038] Based on the unit direction vector, the first normal vector, and the first distance, the corresponding recognition text positioning box in the camera coordinate system is determined.

[0039] The above describes how, by using the vertex coordinates of the image text positioning box corresponding to the text recognition result and the camera's intrinsic parameter matrix, the unit direction vector from the origin of the camera coordinate system to the recognized text positioning box is determined. Based on the unit direction vector, the first normal vector, and the first distance, the recognized text positioning box corresponding to the image text positioning box in the camera coordinate system is determined. This can accurately locate the position of the text in the camera coordinate system, further improving the positioning accuracy of the fusion display between virtual and reality, as well as the accuracy of the fusion display of virtual and reality.

[0040] Furthermore, determining the text plane coordinate system based on the identified text positioning box includes:

[0041] The origin of the text plane coordinate system is determined based on the first vertex of the identified text positioning box;

[0042] The third direction axis of the text plane coordinate system is determined by the negative value of the first normal vector of the text plane;

[0043] The first direction axis of the text plane coordinate system is determined based on the mean of the direction vectors of the first and third sides of the identified text positioning box.

[0044] The second direction axis of the text plane coordinate system is determined based on the cross product of the third direction axis and the first direction axis;

[0045] The text plane coordinate system is defined by the origin, the first direction axis, the second direction axis, and the third direction axis.

[0046] As mentioned above, by identifying the vertices and the distribution of each side of the text positioning box, the text plane coordinate system can be accurately determined. This ensures accurate optimization of the text positioning boxes corresponding to each text recognition result, reduces the distortion of the content displayed on the screen relative to the actual content observed by the user, and improves the user experience.

[0047] Furthermore, determining the tracking target location box corresponding to the identified text location box in the text plane coordinate system includes:

[0048] The first vertex of the tracking target positioning box is determined based on the origin of the text plane coordinate system;

[0049] The width of the tracking target positioning box is determined based on the average projection length of the first and third sides of the identified text positioning box on the first direction axis of the text plane coordinate system.

[0050] The height of the tracking target positioning box is determined based on the average projection length of the second and fourth sides of the identified text positioning box on the second direction axis of the text plane coordinate system.

[0051] The above-mentioned method of accurately determining the tracking target positioning box by using the origin of the text plane coordinate system and the side lengths of the identified text positioning box can improve the positioning accuracy of image tracking of the first image and effectively improve the display effect.

[0052] In a second aspect, embodiments of this application provide a display control device based on an augmented reality device, the augmented reality device being equipped with a camera and a screen, and the display control device including an image acquisition module, an image conversion module, a target tracking module, a position determination module, and a display control module, wherein:

[0053] The image acquisition module is used to acquire a first image obtained by the camera capturing an image of the text plane;

[0054] The image conversion module is used to convert the second image into a tracking target image according to a preset homography matrix and the target image size. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is the first image or an image captured by the augmented reality device before the first image and subjected to text recognition. The homography matrix records the mapping relationship used to convert the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system.

[0055] The target tracking module is used to perform feature point matching between the tracked target image and the first image to obtain a target transformation matrix. The target transformation matrix records a mapping relationship for transforming the tracked target image from the text plane coordinate system to the camera coordinate system.

[0056] The position determination module is used to determine the target tracking position of the tracked target image in the camera coordinate system according to the target transformation matrix;

[0057] The display control module is used to display a preset image on the screen according to the target tracking position.

[0058] This application embodiment acquires a first image obtained by a camera capturing an image of a text plane. A second image is then transformed according to a preset homography matrix and the target image size to obtain a tracking target image. Feature point matching is performed between the tracking target image and the first image to obtain a target transformation matrix. The target tracking position of the tracking target image in the camera coordinate system is determined based on the target transformation matrix, and a preset image is displayed on the screen according to the target tracking position. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition. The homography matrix can be used to transform the text recognition result... The corresponding text image is transformed from the camera coordinate system to the text plane coordinate system, so that the tracking target image can correctly reflect the result of transforming the text recognition result of the second image to the text plane coordinate system. The target transformation matrix obtained by matching the feature points of the tracking target image and the first image can accurately transform the tracking target image to the camera coordinate system, and can accurately determine the target tracking position corresponding to the tracking target image in the camera coordinate system. When the user wears the augmented reality device, the image displayed on the screen observed by the user can be correctly superimposed on the real text object corresponding to the text recognition result, so that the information related to the text recognition result can be correctly superimposed on the real text object for display, further improving the user experience of wearing the augmented reality device.

[0059] In a third aspect, embodiments of this application provide a display control device based on an augmented reality device, including: a memory and one or more processors;

[0060] The memory is used to store one or more programs;

[0061] When the one or more programs are executed by the one or more processors, the one or more processors implement the display control method based on the augmented reality device as described in the first aspect.

[0062] In a fourth aspect, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the display control method based on an augmented reality device as described in the first aspect. Attached Figure Description

[0063] Figure 1 This is a flowchart of a display control method based on an augmented reality device provided in an embodiment of this application;

[0064] Figure 2 This is a flowchart of another display control method based on an augmented reality device provided in an embodiment of this application;

[0065] Figure 3 This is a schematic diagram of an image text positioning box display in a first image provided by an embodiment of this application;

[0066] Figure 4 This is a schematic diagram of a text recognition positioning box from a top-down view of a text plane, provided in an embodiment of this application.

[0067] Figure 5 This is a schematic diagram of a text plane coordinate system from a top-down view of the text plane, provided in an embodiment of this application.

[0068] Figure 6 This is a schematic diagram of a tracking target positioning frame from a top-down view of a text plane, provided in an embodiment of this application.

[0069] Figure 7 This is a schematic diagram of a target image tracking embodiment provided in this application;

[0070] Figure 8 This is a schematic diagram illustrating the display effect of a preset image provided in an embodiment of this application;

[0071] Figure 9 This is a schematic diagram of the structure of a display control device based on an augmented reality device provided in an embodiment of this application;

[0072] Figure 10This is a schematic diagram of the structure of a display control device based on an augmented reality device provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0074] Existing augmented reality (AR) devices based on text recognition typically perform text recognition on captured images, determine the relevant information to be displayed based on the recognition results, and then display this information on the device's screen. However, the displayed result is spatially separated from the actual text object. Users need to confirm the information related to the text recognition results on the screen and then locate the information in reality. For example, when applying AR devices to teaching grading scenarios, a grading system can be configured within the AR device. Users can wear the AR device and use its camera to photograph the math problems to be graded on paper. The photographed image information is then input into the grading system. The grading system uses text recognition technology to identify the numerical equations in the image, grades the problems based on the identified equations, determines the grading result, and finally displays the grading result on the AR device's screen. Because the correction results are displayed directly on the screen, there may be discrepancies between the correction results observed by the user on the screen and the actual numerical formulas observed on the screen. The correction results displayed in the augmented reality device are spatially separated from the actual numerical formulas. This requires the user to manually match the positions of the correction results observed on the screen with the actual numerical formulas observed on the screen. Matching the text recognition results with the real text objects consumes a significant amount of the user's time, severely impacting the user experience of wearing the augmented reality device.

[0075] Based on this, this application provides a display control method based on an augmented reality device to solve the technical problem in existing display solutions where the information related to text recognition results displayed on the augmented reality device is spatially separated from the actual text object, affecting the user experience. By combining the augmented reality device with text recognition technology, the information related to the text recognition results is displayed on the augmented reality device, allowing it to be correctly superimposed on the actual text plane. When the user wears the augmented reality device, the image displayed on the screen can be correctly superimposed on the actual text object corresponding to the text recognition result, enabling the information related to the text recognition results to be correctly superimposed on the actual text object for display, further improving the user experience when wearing the augmented reality device.

[0076] For ease of description, the following definitions are used in this embodiment: bolded English letters represent vectors or matrices, and unbolded English letters represent scalar numbers; for the position of a three-dimensional point in space, a 3×1 large vector is represented by uppercase English letters, such as X=[X,Y,Z] T The position of a two-dimensional pixel in the image is represented by a 2×1 vector using lowercase English letters, such as x = [x, y]. T ,in,[] T This indicates the transpose of a matrix, and the superscript of the letter indicates the coordinate system of the point. This indicates a transformation from a normal coordinate system to a homogeneous coordinate system, such as... For example, adding an element of size 1 to the end of a vector. This indicates a transformation from a homogeneous coordinate system to a normal coordinate system. The inverse process removes the last element of the input vector, such as...

[0077] R is a 3×3 rotation matrix, t is a 3×1 translation vector; T represents a 3×4 spatial transformation matrix, indicating the spatial relationship between two coordinate systems, also called the extrinsic parameter matrix, and has the following:

[0078] T = [Rt]

[0079]

[0080] In this system, the lower left subscript of the letter indicates the source coordinate system, and the upper left subscript indicates the target coordinate system, such as... Represents the rotation matrix for the transformation from coordinate system {a} to coordinate system {b}; This represents the displacement vector from the {a} coordinate system to the {b} coordinate system. This represents a spatial transformation from the {a} coordinate system to the {b} coordinate system.

[0081] K is a 3×3 intrinsic parameter matrix in the pinhole imaging model, which projects a 3D point X in space onto the 2D position x of image I, as follows:

[0082]

[0083] Where λ is the scale factor, which can be any scalar; fx is the camera focal length in the x-direction (in pixels); fy is the camera focal length in the y-direction (in pixels); cx is the pixel distance of the origin in the x-direction; cy is the pixel distance of the origin in the y-direction, i.e., cx; cy is the aperture center position of the camera; and I(i) represents the pixel value of the i-th pixel in the image.

[0084] Suppose there exists a three-dimensional point X = [X, Y, 1] on the XY plane in the world coordinate system. T (Assuming Z = 1), its projected pixel coordinates in the image are x = [u, v] T There is a relationship between them:

[0085]

[0086] Define R = [r1, r2, r3], r i If i∈{1,2,3} represents the i-th column of the rotation matrix, then the homography matrix H is:

[0087] H=K[r1,r2,t]=[Kr1,Kr2,Kt]=[h1,h2,h3]

[0088] H is a 3×3 matrix, h i Let be the i-th column of the matrix. Based on the above formula, we can obtain:

[0089]

[0090] The homography matrix H can map a 3D point X on a plane to a 2D point x in an image, since the 3D point X = [X, Y, 1] on the plane. T One dimension is fixed (Z=1), and X can be considered as a point x′ on the XY two-dimensional plane, then... Alternatively, the homography matrix H can be considered to map a point x′ on a two-dimensional plane to a two-dimensional point x in an image, i.e., a two-dimensional to two-dimensional point-to-point mapping:

[0091]

[0092] And there exists an inverse matrix H -1 ,have:

[0093]

[0094] If H1 and H2 are both homography matrices, then their product H3 = H1H2 is also a homography matrix. Suppose there exists a template image and a target image, and we need to know the position of the template image in the target image. Let the coordinates of any pixel in the template image be x′, and the corresponding pixel coordinates in the target image be x. Then there exists a homography matrix H, with λ... The homography matrix H can then be used as a representation of the localization result of the template image on the target image. That is, define the transformation operation warp(I,H,w,h), where I′=warp(I,H,w,h). For each pixel x in the original image I, according to λ... Map the coordinates to pixel x′ of graph I′. The new mapped graph I′ has a length of w and a width of h, in pixels.

[0095] Figure 1 A flowchart of a display control method based on an augmented reality device provided in this application embodiment is given. The display control method based on an augmented reality device provided in this application embodiment can be executed by a display control device based on an augmented reality device. The display control device based on an augmented reality device can be implemented by hardware and / or software and integrated into the display control device based on the augmented reality device.

[0096] The following description uses an augmented reality device-based display control device as an example to illustrate the execution of an augmented reality device-based display control method. (Reference) Figure 1 The display control method based on augmented reality devices includes:

[0097] S101: Obtain the first image obtained by the camera capturing the text plane.

[0098] The augmented reality (AR) device provided in this solution can be AR glasses, AR helmets, etc. The AR device is equipped with a camera and a screen, with the camera facing forward so that its field of view is similar to or the same as the user's field of vision when wearing the AR device. The camera can capture images and perform relevant processing on the captured images (e.g., text recognition, image tracking). The screen in the AR setup is a translucent (e.g., semi-transparent) screen. The AR device can display patterns on the screen, which the user can observe. The user can also observe the real world through the screen, allowing the pattern on the screen to be superimposed onto the real world. After virtual-real calibration, the AR device can determine the position on the screen where the pattern needs to be superimposed on the real world, based on the mapping relationship between the real world and virtual space. The rendering camera then renders the corresponding pattern at that position, and the pattern observed by the user is superimposed onto the corresponding position in the real world.

[0099] In this solution, the camera is used to capture images corresponding to the real scene, and the rendering camera is used to simulate the camera's perspective and render images from that perspective, such as rendering and displaying a preset image on the screen based on rendering position information. The rendering camera can be understood as a virtual device or simulator used to simulate the camera's perspective during the graphics rendering process. In a virtual scene, the rendering camera can capture and render the final two-dimensional image. By adjusting parameters such as the position, orientation, and focal length of the rendering camera, the perspective, perspective distortion, and depth of field effects of the rendered image can be controlled, thereby rendering a two-dimensional image that conforms to visual habits and displaying it on the screen. The rendering camera can use orthographic projection or perspective projection to render and project images. Orthographic projection can maintain the proportions of objects, while perspective projection can simulate the visual effect of the human eye, producing perspective distortion where objects appear larger when closer and smaller when farther away, making the image more three-dimensional and realistic. Optionally, the rendering camera provided in this solution can use perspective projection so that when the projected image is superimposed on the display, the displayed image blends more naturally with the real world, improving the user experience of wearing augmented reality devices.

[0100] For example, when it is necessary to display information related to text recognition results using an augmented reality device, the user can wear the augmented reality device, with their field of vision facing the text plane, and use the camera on the augmented reality device to take pictures, continuously acquiring images of the text plane captured by the camera. The currently acquired image is then used as the first image, and the display control method provided in this application is executed. In subsequent display control processes, the most recently acquired image is used as the first image; the first image in the current display control process is no longer used as the first image in subsequent display control processes. The text plane can be understood as the plane in the real world where the text to be recognized is located. For example, when text recognition is needed on a textbook or desk, the plane where the textbook or desk is located is the text plane.

[0101] S102: Based on the preset homography matrix and the target image size, the second image is transformed to obtain the tracking target image.

[0102] The target image size provided by this solution can be determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by the augmented reality device before the first image and for which text recognition has been performed.

[0103] Optionally, text recognition can be performed on each acquired first image (in which case the second image is the latest acquired first image), and target image tracking and image display can be performed based on the latest text recognition result. Alternatively, text recognition can be omitted for each first image. After text recognition is performed on the current first image, the second image corresponding to the current display control flow becomes the current first image. When the display control flow is subsequently executed for a new image (i.e., a new first image), before text recognition is performed on the new image, the current first image will participate in the subsequent display control flow as the second image (i.e., the image captured before the new first image and for which text recognition has been performed).

[0104] The homography matrix provided by this solution records the mapping relationship used to transform the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system. The text plane coordinate system is determined on the text plane based on the text recognition result. For example, the first and second direction axes (e.g., the X-axis and Y-axis) of the text plane coordinate system are on the text plane, and the third direction axis (e.g., the Z-axis) is perpendicular to the text plane.

[0105] For example, after obtaining the first image, the text image corresponding to the text recognition result of the second image is transformed from the camera coordinate system to the text plane coordinate system according to the preset homography matrix and the target image size, so as to obtain the tracking target image, wherein the size of the tracking target image is the target image size.

[0106] S103: Perform feature point matching between the target image and the first image to obtain the target transformation matrix.

[0107] For example, after determining the target image to be tracked, feature points of both the target image and the first image are obtained. Feature point matching is then performed between the target image and the first image using these feature points, and a target transformation matrix is ​​calculated based on the matching results. Optionally, feature points can be extracted using feature point detection algorithms such as ORB, SIFT, or SURF. The target transformation matrix records the mapping relationship used to transform the target image from the text plane coordinate system to the camera coordinate system.

[0108] In one embodiment, the target transformation matrix can be a perspective transformation matrix calculated based on feature point matching pairs formed by a set number (e.g., four or more) of feature points from the tracking target image and feature points from the first image. Optionally, the target transformation matrix can be calculated based on a preset transformation matrix acquisition function (e.g., the findHomography function). The perspective transformation matrix can be used to realize perspective transformation of an image under different viewpoints, projecting the image from one viewpoint plane to another. For example, the transformation matrix maps pixels in the original image to corresponding positions in the target image, thereby achieving perspective transformation of the image.

[0109] Optionally, after acquiring the first image, the target image can be tracked on the first image based on the set image tracking method (planar object tracking algorithm), and the target transformation matrix generated during the tracking process can be obtained by transforming the target image from the text plane coordinate system to the camera coordinate system.

[0110] Optionally, the image tracking method provided in this scheme can be an image matching method based on ORB (Oriented FAST and Rotated BRIEF, a fast feature point extraction and description algorithm), SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), etc.; it can be a matching method based on pixel motion such as KLT (Kanade-Lucas-Tomasi, corner tracking algorithm), ESM (Efficient Second Order Minimization), IC (Inverse Compositional), etc.; or it can be a method based on deep learning networks such as HomographyNet (homography matrix network) and UDH (Unsupervised Deep Homography).

[0111] S104: Determine the target tracking position in the camera coordinate system corresponding to the target image based on the target transformation matrix.

[0112] For example, the target tracking position of the target image in the camera coordinate system is determined based on the target transformation matrix determined above. For instance, the positioning box corresponding to the target image is transformed from the text plane coordinate system to the camera coordinate system based on the target transformation matrix to obtain the target tracking position of the target image in the camera coordinate system.

[0113] S105: Display a preset image on the screen based on the target tracking position.

[0114] After determining the target tracking position, a preset image can be displayed on the screen based on that position. Optionally, the preset image can be determined according to display requirements. For example, the preset image can be a positioning box used to surround the recognized content, or it can indicate the analysis results of the text content corresponding to the text recognition result. The content can be text, formulas, questions, etc., and the analysis results can indicate whether the text content is correct (e.g., whether the formula or equation corresponding to the text content is correct, or whether the answer to the question is correct).

[0115] Optionally, after a preset image is displayed on the screen, the superimposed effect of the preset image and the real world observed by the user wearing the augmented reality device can be either the preset image overlapping with the real text object or offset from the real text object. Optionally, the display of analysis results can be achieved through the display style of positioning boxes, such as using positioning boxes of different colors or line shapes to indicate different analysis results (e.g., using a red positioning box to indicate an incorrect analysis result, and a green positioning box to indicate a correct analysis result). The preset image can also be an identifier used to indicate the analysis result of the text content corresponding to the text recognition result, for example, using an "X" identifier to indicate an incorrect analysis result, and using a "Check" identifier to indicate a correct analysis result.

[0116] The above describes a process where a first image is captured by a camera on a text plane. A second image is then transformed based on a preset homography matrix and the target image size to obtain a tracking target image. Feature point matching is performed between the tracking target image and the first image to obtain a target transformation matrix. The target tracking position in the camera coordinate system is determined based on this matrix, and a preset image is displayed on the screen according to this target tracking position. The target image size is determined based on the size of the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition. The homography matrix can be used to transform the text recognition result into a target image. The corresponding text image is transformed from the camera coordinate system to the text plane coordinate system, so that the tracking target image can correctly reflect the result of transforming the text recognition result of the second image to the text plane coordinate system. The target transformation matrix obtained by matching the feature points of the tracking target image and the first image can accurately transform the tracking target image to the camera coordinate system, and can accurately determine the target tracking position corresponding to the tracking target image in the camera coordinate system. When the user wears the augmented reality device, the image displayed on the screen observed by the user can be correctly superimposed on the real text object corresponding to the text recognition result, so that the information related to the text recognition result can be correctly superimposed on the real text object for display, further improving the user experience of wearing the augmented reality device.

[0117] Based on the above embodiments, Figure 2 A flowchart of another display control method based on an augmented reality device provided in this application is given. This display control method based on an augmented reality device is a concretization of the above-described display control method based on an augmented reality device. (Reference) Figure 2 The display control method based on augmented reality devices includes:

[0118] S201: Acquire the first image obtained by the augmented reality device taking an image of the text plane.

[0119] S202: If the text recognition triggering condition is met, perform text recognition on the first image to obtain the text recognition result.

[0120] S203: Determine the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system.

[0121] The text recognition triggering conditions provided in this solution can be understood as the conditions that trigger text recognition of the first image. These conditions can be determined based on the user's text recognition triggering operation, the captured first image, or a set time interval, such as the time interval between the current time and the last text recognition trigger (the text recognition triggering condition is considered met when the time interval reaches a set time threshold). The user's text recognition triggering operation can be triggered via an interactive button on the augmented reality device or via an external device connected to the augmented reality device (e.g., mouse, keyboard, remote control, smart mobile device, etc.). When determining whether the text recognition triggering condition is met based on the captured first image, image recognition can be performed on the first image (or image recognition can be performed on the captured first image at set time intervals). If a set gesture, posture, or text content is recognized, the text recognition triggering condition is considered met.

[0122] For example, when the text recognition trigger condition is met, text recognition processing is performed on the acquired first image to obtain a text recognition result. Optionally, the text recognition result provided by this solution may include the image text obtained from the text recognition processing and the image text positioning box (the image text positioning box can be represented by the coordinates of the four vertices in the first image). The image text positioning box can be understood as a positioning box on the first image that encloses the recognized image text. Optionally, the text recognition processing on the first image may be performed on a set text type, such as performing text recognition processing on words, equations, or questions to obtain a text recognition result where the text content is words, equations, or questions.

[0123] Optionally, the text recognition processing function for the first image can be deployed locally or in the cloud. For example, the first image can be uploaded to the cloud, and text recognition processing can be performed on the first image in the cloud to obtain the text recognition result. By deploying the text recognition processing function in the cloud, the problem of excessive computational load in text recognition processing and low efficiency of local text recognition processing, which affects the efficiency of determining the tracking target bounding box, can be reduced.

[0124] In one embodiment, the number of text recognition results obtained by performing text recognition processing on the first image corresponds to the number of characters, equations, and questions in the first image. When there are no characters, equations, or questions in the first image, the text recognition processing of the first image is considered as no target text content is detected, and the current display control flow ends until the first image is acquired again and the text recognition trigger condition is met.

[0125] like Figure 3 A schematic diagram showing an image text positioning box in a first image is provided. Figure 3The first image is a rectangle. A text plane can be observed within the first image (in the image, the text plane appears as a rectangle, narrower at the top and wider at the bottom, due to perspective). Two text contents, "Hello" and "1+1=2", exist within this text plane. After processing the corresponding text in the first image, two text recognition results can be obtained. The image text corresponding to these two results is "Hello" and "1+1=2", respectively. The vertex coordinates of the image text location boxes are { C x i，0 , C x i,1 , C x i,2 , C x i，3}, i∈{0,1,...,N-1}, where N is the number of recognized text results. Figure 3 In this context, N is 2, and the image text "Hello" corresponds to the image text positioning box { C x 0,0 , C x 0,1 , C x 0,2 , C x 0,3 The image text "1+1=2" corresponds to the image text positioning box {}. C x 1，0 , C x 1，1 , C x 1，2 , C x 1，3}

[0126] After obtaining the text recognition results, for each text recognition result, the corresponding image text localization box is transformed from the camera coordinate system to the text plane coordinate system, resulting in the tracking target localization box corresponding to the image text localization box in the text plane coordinate system. The tracking target localization box can be understood as the localization box of the image text localization box transformed from the camera coordinate system to the text plane coordinate system.

[0127] This solution obtains text recognition results by performing text recognition on the first image when the text recognition trigger condition is met. The corresponding image text positioning box is then transformed into a text plane coordinate system to obtain the tracking target positioning box. Subsequent images can be tracked based on this tracking target positioning box, eliminating the need for text recognition on every single first image. This reduces the frequency of text recognition after each first image acquisition, lowers resource consumption, and further improves the efficiency of virtual-to-real-world positioning switching while maintaining the accuracy of the merged display. For example, when the text recognition processing function for the first image can be deployed in the cloud, text recognition is only performed on the first image when the text recognition trigger condition is met, improving the real-time performance of display control while maintaining display control effectiveness.

[0128] In one possible embodiment, the display control method based on augmented reality devices provided in this solution determines the tracking target positioning box corresponding to the image text positioning box of the text recognition result in the text plane coordinate system. This can be achieved by:

[0129] S2031: Determine the first normal vector of the text plane corresponding to the text recognition result and the first distance from the text plane to the camera.

[0130] S2032: Based on the image text localization box corresponding to the text recognition result, the camera's intrinsic parameter matrix, the first normal vector, and the first distance, determine the recognized text localization box in the camera coordinate system.

[0131] S2033: Determine the text plane coordinate system based on the identified text positioning box, and determine the tracking target positioning box corresponding to the identified text positioning box in the text plane coordinate system.

[0132] For example, after obtaining the text recognition results, for each text recognition result, the text recognition result is input into a set plane detection algorithm. The plane detection algorithm determines the first normal vector of the text plane corresponding to the text recognition result and the first distance from the text plane to the camera. Optionally, the plane detection algorithm provided in this solution can be a texture-based plane detection algorithm, such as the RANSAC plane fitting algorithm, or a deep learning network-based plane detection algorithm, such as Plane R-CNN, Plane RecNet, and other deep learning networks.

[0133] After determining the first normal vector and the first distance, based on the image text localization box corresponding to the text recognition result, the camera's intrinsic parameter matrix (which can be set at the factory or determined during virtual-real calibration or debugging), the first normal vector, and the first distance, the image text localization box is transformed from the text plane coordinate system to the camera coordinate system. The corresponding recognized text localization box in the camera coordinate system is then determined. The recognized text localization box can be understood as the localization box of the image text localization box in the camera coordinate system. Specifically, the tracking target localization box is a two-dimensional localization box on the text plane, the image text localization box is a two-dimensional localization box on the first image, and the recognized text localization box is a three-dimensional localization box in the camera coordinate system.

[0134] Furthermore, for each text recognition result corresponding to the recognition text location box, a text plane coordinate system is determined based on the recognition text location box on the text plane. The first and second direction axes (e.g., X-axis and Y-axis) of the text plane coordinate system are on the text plane, the third direction axis (e.g., Z-axis) is perpendicular to the text plane, and the origin of the text plane coordinate system is at a vertex of the recognition text location box, for example, the origin of the text plane coordinate system is at the vertex of the upper left corner of the recognition text location box.

[0135] Furthermore, after determining the text plane coordinate system corresponding to each text recognition result, the recognized text positioning box is transformed from the camera coordinate system to the text plane coordinate system in order to determine the tracking target positioning box corresponding to each recognized text positioning box in the corresponding text plane coordinate system.

[0136] This solution determines the recognized text positioning box in the camera coordinate system by using the first normal vector of the text plane corresponding to the text recognition result, the first distance from the text plane to the camera, and the intrinsic parameter matrix of the camera. It also determines the text plane coordinate system of each recognized text positioning box and the tracking target positioning box corresponding to the recognized text positioning box in the text plane coordinate system. The shape of the obtained tracking target positioning box is closer to the shape of the actual text area in the first image. Based on the tracking target positioning box, the target image can be tracked more accurately on the first image, improving the positioning accuracy of the fusion display between virtual and reality.

[0137] In one possible embodiment, the display control method based on augmented reality devices provided by this solution determines the recognized text positioning box in the camera coordinate system according to the image text positioning box corresponding to the text recognition result, the camera's intrinsic parameter matrix, the first normal vector, and the first distance. This can be achieved by: determining the unit direction vector from the origin of the camera coordinate system to the recognized text positioning box based on the vertex coordinates of the positioning box of the image text positioning box corresponding to the text recognition result and the camera's intrinsic parameter matrix; and determining the recognized text positioning box in the camera coordinate system based on the unit direction vector, the first normal vector, and the first distance.

[0138] For example, the vertex coordinates of the image text localization box corresponding to the text recognition result are transformed to a homogeneous coordinate system. The product of the vertex coordinates of the image text localization box in the homogeneous coordinate system and the inverse matrix of the camera's intrinsic parameter matrix is ​​used as the unit direction vector from the origin of the camera coordinate system to the recognized text localization box. For example, the unit direction vector corresponding to the i-th text recognition result can be determined by the following formula:

[0139]

[0140]

[0141] in, c D is the direction vector from the origin of the camera coordinate system to the recognition text positioning box. c K -1 This is the inverse of the camera's intrinsic parameter matrix. C x = { C x i，0 , C x i，1 , C x i，2 , C x i，3 Let} be the vertex coordinates of the i-th image text positioning box. Let be the unit direction vector from the origin of the camera coordinate system to the recognized text location box, and ‖·‖ be the L2 norm sign. Optionally, the camera's intrinsic parameter matrix can be represented as:

[0142]

[0143] Where fx1 is the camera focal length in the x-direction (in pixels), fy1 is the camera focal length in the y-direction (in pixels), cx1 is the distance in pixels that the camera's origin is shifted in the x-direction, and cy1 is the distance in pixels that the camera's origin is shifted in the y-direction. That is, (cx1, cy1) is the center position of the camera's aperture.

[0144] Furthermore, based on the product of the negative value of the unit direction vector and the ratio of the dot product of the unit direction vector and the first normal vector, the corresponding text localization box in the camera coordinate system is determined. That is, the four vertices of the text localization box are the intersection points of the directions of the unit direction vectors from the origin of the camera coordinate system to the four vertices of the text localization box and the text plane. For example, the text localization box corresponding to the i-th text recognition result can be determined by the following formula:

[0145]

[0146] in, C X = { C X i，0 , C X i，1 , C X i，2 , C X i，3 Let} be the vertex coordinates of the i-th recognized text bounding box. C d is the first distance. This is the first normal vector. This scheme determines the unit direction vector from the origin of the camera coordinate system to the recognized text positioning box by using the vertex coordinates of the image text positioning box corresponding to the text recognition result and the camera's intrinsic parameter matrix. Based on the unit direction vector, the first normal vector, and the first distance, it determines the corresponding recognized text positioning box in the camera coordinate system. This accurately locates the text position in the camera coordinate system, further improving the positioning accuracy of the fusion display between virtual and reality, and enhancing the accuracy of the fusion display of virtual and reality images.

[0147] like Figure 4 A schematic diagram of a text recognition localization box from a top-down view of a text plane is provided, wherein the text in the image is "Hello" and the corresponding text recognition localization box is { C X 0,0 , C X 0,1 , C X 0,2 , C X 0,3}, the corresponding text location box for the image text "1+1=2" is { C X 1，0 , C X 1，1 , C X 1，2 , C X 1，3Because of the angle between the camera and the text plane, the text recognition bounding box is irregular on the text plane due to perspective geometry. If the target image is tracked and displayed on the screen based on this irregular bounding box, the displayed content will be severely distorted relative to what the user observes, affecting the user experience. Therefore, this solution optimizes the text recognition bounding box to obtain a tracking target bounding box. This box is then used to track the target image and display the content on the screen, reducing the distortion of the displayed content relative to the user's actual observation and improving the user experience.

[0148] In one possible embodiment, the display control method based on augmented reality devices provided by this solution determines the text plane coordinate system based on the identified text positioning box. This can be achieved by: determining the origin of the text plane coordinate system based on the first vertex of the identified text positioning box; determining the third direction axis of the text plane coordinate system based on the negative value of the first normal vector of the text plane; determining the first direction axis of the text plane coordinate system based on the mean of the direction vectors of the first and third sides opposite to the identified text positioning box; determining the second direction axis of the text plane coordinate system based on the cross product of the third direction axis and the first direction axis; and determining the text plane coordinate system composed of the origin, the first direction axis, the second direction axis, and the third direction axis.

[0149] For example, for each text recognition result, the first vertex (top left corner) of the recognized text location box is taken as the origin, the negative value of the first normal vector of the text plane is taken as the third direction axis (Z-axis), the mean of the direction vectors of the first side (top) and the third side (bottom) of the recognized text location box is taken as the first direction axis (X-axis), and the cross product of the first direction axis and the third direction axis is taken as the second direction axis (Y-axis). After determining the origin, first direction axis, second direction axis, and third direction axis of the text plane coordinate system, the text plane coordinate system formed by the origin, first direction axis, second direction axis, and third direction axis of the text plane coordinate system is further determined, thereby creating the text plane coordinate system corresponding to each text recognition result.

[0150] This solution can accurately determine the text plane coordinate system by identifying the vertices and the distribution of each edge of the text positioning box. It can ensure accurate optimization of the text positioning box corresponding to each text recognition result, reduce the distortion of the content displayed on the screen relative to the actual content observed by the user, and improve the user experience.

[0151] like Figure 5 The diagram illustrates a text plane coordinate system from a top-down view of the text plane. Taking the text "Hello" as an example, it shows the corresponding text location boxes for the text recognition results. C X = {C X i，0 , C X i，1 , C X i，2 , C X i，3 The direction vectors of the four sides of the recognized text bounding box corresponding to the i-th text recognition result are calculated using the following formula:

[0152] C V i,0 = C X i，1 - C X i,0

[0153] C V i，1 = C X i，2 - C X i，1

[0154] C V i,2 = C X i，2 - C X i，3

[0155] C V i，3 = C X i，3 - C X i,0

[0156] in, C V i,0 - C V i,3 These are the direction vectors for the four sides of the text positioning box: top, right, bottom, and left.

[0157] Define a text plane coordinate system {O} for each text recognition result. c X i,0 Let O be the origin of the text plane coordinate system, and let the third direction axis of the text plane coordinate system be... The negative of the first normal vector of the text plane: First direction axis To identify the mean of the direction vectors of the top and bottom edges of the text positioning box, the second direction axis Perpendicular to the first direction axis and the third direction axis, it is obtained by the cross product of the third direction axis and the first direction axis. The direction axes of the text plane coordinate system corresponding to the i-th text recognition result can be determined by the following formula:

[0158]

[0159] in, C V i,0 and c V i,2 These are the direction vectors of the top and bottom edges of the identified text positioning box, respectively.

[0160] In one possible embodiment, the display control method based on augmented reality devices provided by this solution determines the tracking target positioning box corresponding to the identified text positioning box in the text plane coordinate system by: determining the first vertex of the tracking target positioning box according to the origin of the text plane coordinate system; determining the width of the tracking target positioning box according to the average projection length of the first and third sides of the identified text positioning box on the first direction axis of the text plane coordinate system; and determining the height of the tracking target positioning box according to the average projection length of the second and fourth sides of the identified text positioning box on the second direction axis of the text plane coordinate system.

[0161] For example, the origin of the text plane coordinate system is used as the first vertex of the optimized tracking target positioning box. The average projection length of the first (top) and third (bottom) sides of the identified text positioning box on the first direction axis (X-axis) of the text plane coordinate system is used as the width of the tracking target positioning box. This average projection length is the average of the sum of the projection lengths of the first side and the third side of the text positioning box on the first direction axis of the text plane coordinate system. The average projection length of the second (right) and fourth (left) sides of the identified text positioning box on the second direction axis (Y-axis) of the text plane coordinate system is used as the height of the tracking target positioning box. This average projection length is the average of the sum of the projection lengths of the second and fourth sides of the text positioning box on the second direction axis of the text plane coordinate system. This solution accurately determines the tracking target positioning box by using the origin of the text plane coordinate system and the side lengths of the identified text positioning box, improving the accuracy of image tracking of the first image and effectively improving the display effect.

[0162] like Figure 6 A schematic diagram of the tracking target localization box from a top-down view of the text plane is provided, where the dashed box represents the recognized text localization box, and the solid box represents the tracking target localization box optimized based on the recognized text localization box. Optionally, the width and height of the tracking target localization box corresponding to the i-th text recognition result can be determined by the following formula:

[0163] wm i =|| C VX i ||

[0164]

[0165] Among them, wm i To track the width of the target bounding box, hm i To track the height of the target bounding box, C X i,0 With the origin as the point, The directional length is wm i As the top of the positioning frame, The direction and height are hm i As the sides of the positioning box, a new tracking target positioning box is formed. The four vertices of the tracking target positioning box are located in the corresponding text plane coordinate system {O} as { O X i，0 , O X i，1 , O X i，2 , O X i，3}:

[0166] O X i，0 =[0,0,0] T

[0167] O X i，1 =[wm i ,0,0] T

[0168] O X i，2 =[wm i , hm i ,0] T

[0169] O X i，3 =[0,hm i ,0] T

[0170] S204: Determine the width of the target image based on the first side length of the image text positioning box.

[0171] S205: Determine the height of the target image by multiplying the ratio of the height to the width of the target positioning box by the width of the target image.

[0172] In this scheme, the image text positioning box is determined based on the text recognition result of the second image, and the tracking target positioning box is obtained by transforming the image text positioning box from the text plane coordinate system to the text plane coordinate system.

[0173] For example, for each text recognition result obtained by performing text recognition on the second image, the first side length of the image text positioning box corresponding to the text recognition result is taken as the target image width corresponding to the target image size, and the product of the ratio of the height and width of the tracking target positioning box corresponding to the text recognition result in the text plane coordinate system and the target image width is taken as the target image height corresponding to the target image size. Optionally, the target image width and target image height (in pixels) corresponding to the target image size of the i-th text recognition result can be determined by the following formula:

[0174] wp i = C x i,1 - C x i,0

[0175]

[0176] Among them, wp i For the target image width, hp i The height of the target image. C x i,0 The coordinates of the top-left corner of the image text positioning box. c x i,1 The coordinates of the top right corner of the image text positioning box. C x i,1 - C x i,0 hm is the first side length of the image text positioning box. i To track the height of the target bounding box, wm i To track the width of the target positioning box, this method determines the target image width by the first side length of the image text positioning box, and determines the target image height based on the height and width of the tracking target positioning box and the target image width. This allows for accurate determination of the target image size and improves the accuracy of acquiring and tracking the target image.

[0177] S206: Determine the homography matrix based on the tracking target location box and the camera's intrinsic parameter matrix corresponding to the text recognition result of the second image in the text plane coordinate system, as well as the preset unit conversion ratio between image distance units and spatial distance units.

[0178] For example, for each text recognition result obtained by performing text recognition on the second image, the homography matrix is ​​determined based on the tracking target localization box corresponding to the text recognition result, the camera's intrinsic parameter matrix, and the unit conversion ratio between image distance units and spatial distance units. The unit conversion ratio between image distance units and spatial distance units can be determined based on the ratio of image distance units to spatial distance units. Image distance units represent distance units on an image, and spatial distance units represent distance units in space. For example, when the image distance unit is pixels and the spatial distance unit is meters, the corresponding unit conversion ratio is pixels / meter. The unit conversion ratio can be used to convert values ​​in image distance units to values ​​in spatial distance units, for example, converting the image distance units (e.g., pixels) of the first or second image to real-world spatial distance units (e.g., meters). Optionally, the homography matrix corresponding to the i-th text recognition result can be determined using the following formula:

[0179]

[0180] α i =wm i / wp i

[0181] in, O H i It is a homography matrix. C K is the intrinsic parameter matrix of the camera. Let x be the first direction axis of the text plane coordinate system. Let be the second direction axis of the text plane coordinate system. C X i,0 Let α be the origin of the text plane coordinate system. i For unit conversion ratios, wm i To track the width of the target bounding box, wp i The target image width is defined here. This solution accurately determines the homography matrix by tracking the target bounding box, the camera's intrinsic parameter matrix, and the unit conversion ratio, thereby improving the accuracy of acquiring and tracking target images.

[0182] S207: Based on the homography matrix and the target image size, the second image is transformed to obtain the tracking target image.

[0183] like Figure 7 The provided schematic diagram illustrates a method for tracking a target image, where the target image width wp is determined. i Target image height hp i and homography matrix O H i Then, based on the homography matrix and the target image width wp i and target image height hpi The second image is then transformed to obtain the target image.

[0184] Optionally, the tracking target image corresponding to the i-th text recognition result can be determined by the following formula:

[0185]

[0186] Here, warp is the image mapping operation command. o I i For the tracking target image corresponding to the i-th text recognition result, c I OCR For the second image, It is the inverse of the homography matrix.

[0187] S208: Based on the target image being tracked, target image tracking is performed on the first image to obtain the target transformation matrix that transforms the target image from the text plane coordinate system to the camera coordinate system.

[0188] S209: Based on the target transformation matrix, transform the tracking target location box corresponding to the text recognition result of the second image in the text plane coordinate system to the camera coordinate system, and obtain the target tracking position corresponding to the tracking target image in the camera coordinate system.

[0189] For example, after determining the target transformation matrix, for each text recognition result obtained from text recognition on the second image, the target localization box corresponding to each text recognition result is transformed to the camera coordinate system using the target transformation matrix, thus obtaining the target tracking position of the tracking target image in the camera coordinate system. Optionally, the target tracking position corresponding to the i-th text recognition result can be determined by the following formula:

[0190]

[0191] in, c X i For target tracking position, Let be the target transformation matrix. o X i To track the target bounding box, this solution uses a target transformation matrix to convert the target bounding box to the camera coordinate system, accurately obtaining the target tracking position and effectively improving the accuracy of acquiring images of the tracked target.

[0192] S210: Based on the intrinsic and extrinsic parameter matrices of the rendering camera, the target tracking position is transformed from the camera coordinate system to the rendering camera coordinate system to obtain the rendering position information, and a preset image is displayed on the screen based on the rendering position information.

[0193] After the augmented reality device undergoes virtual-real combination calibration, the intrinsic and extrinsic parameter matrices of the rendering camera used to render virtual images can be obtained. The intrinsic and extrinsic parameter matrices of the rendering camera can be used to project spatial points in the camera coordinate system onto the two-dimensional pixels of the augmented reality device screen. This can effectively ensure that the human eye, the two-dimensional pixels of the screen, and the corresponding spatial points in the camera coordinate system are on the same line of sight, so that the virtual image and the real image can be correctly superimposed.

[0194] For example, for each text recognition result obtained by performing text recognition on the second image, based on the intrinsic and extrinsic parameter matrices of the rendering camera, the target tracking position corresponding to the text recognition result is transformed from the camera coordinate system to the rendering camera coordinate system to obtain rendering position information, and a preset image is displayed on the screen according to the rendering position information. For example, the preset image is rendered and displayed at the position corresponding to the rendering position information on the screen, or the preset image is displayed within the range corresponding to the rendering position information on the screen.

[0195] Optionally, the rendering position information corresponding to the i-th text recognition result can be determined by the following formula:

[0196]

[0197] Where λ is the set scale factor, R x i To render position information, R K is the intrinsic parameter matrix of the rendering camera. To render the camera's extrinsic matrix, C X i This method transforms the target tracking position from the camera coordinate system to the rendering camera coordinate system using the intrinsic and extrinsic parameter matrices of the rendering camera to obtain the rendered position information. This allows the preset image to be correctly superimposed onto the corresponding text position in reality, improving the efficiency of positioning switching between virtual and reality, as well as the accuracy of the fused display between virtual and reality. Optionally, the intrinsic parameter matrix of the rendering camera can be represented as:

[0198]

[0199] Where fx2 is the focal length of the rendering camera in the x-direction (in pixels), fy2 is the focal length of the rendering camera in the y-direction (in pixels), cx2 is the pixel distance of the origin of the rendering camera translated in the x-direction, and cy2 is the pixel distance of the origin of the rendering camera translated in the y-direction. That is, (cx2, cy2) is the center position of the aperture of the rendering camera.

[0200] like Figure 8 The provided diagram illustrates a preset image display effect, wherein... Figure 8To illustrate the user's experience through the augmented reality device screen, this solution uses a positioning frame used to enclose text content as a preset image. Here, "1+2×3=1+6=7" represents the text content displayed on the text plane. If the target image is tracked according to the recognized text positioning frame and the preset image K1 is displayed on the screen, in the superimposed display of the screen content and the real world observed by the user, the upper left corner of the text content "1+2×3=1+6=7" tilts to the left, indicating that the text content is tilted relative to the screen. However, the border of the preset image K1 is parallel to the screen, meaning the preset image K1 is "aligned" relative to the screen. It is evident that the tilt of the preset image K1 is not synchronized with the tilt of the text content actually observed by the user. The disconnect between the preset image K1 and the text content actually observed by the user is quite severe, negatively impacting the user experience. This solution utilizes a target positioning bounding box to track the target image and display a preset image K2 on the screen. The tilt direction of the preset image K2 is synchronized with the actual tilt direction of the text content, which is also tilted to the left from the upper left corner. Furthermore, the tilt angle is consistent with the tilt angle of the text content. It can be seen that the tilt of the preset image K2 is synchronized with the tilt of the text content actually observed by the user. The sense of disconnect between the preset image K2 and the text content actually observed by the user is small, which can effectively improve the user's experience when wearing augmented reality devices.

[0201] The above describes a process where a first image is captured by a camera on a text plane. A second image is then transformed based on a preset homography matrix and the target image size to obtain a tracking target image. Feature point matching is performed between the tracking target image and the first image to obtain a target transformation matrix. The target tracking position in the camera coordinate system is determined based on this matrix, and a preset image is displayed on the screen according to this target tracking position. The target image size is determined based on the size of the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition. The homography matrix can be used to transform the text recognition result into a target image. The corresponding text image is transformed from the camera coordinate system to the text plane coordinate system, ensuring that the tracking target image accurately reflects the result of transforming the text recognition result of the second image into the text plane coordinate system. The target transformation matrix obtained based on the feature point matching results of the tracking target image and the first image accurately transforms the tracking target image into the camera coordinate system, precisely determining the target tracking position in the camera coordinate system. When the user wears the augmented reality device, the image displayed on the screen is correctly superimposed on the real text object corresponding to the text recognition result, ensuring that the information related to the text recognition result is correctly superimposed on the real text object for display, further improving the user experience of wearing the augmented reality device. Simultaneously, when the text recognition trigger condition is met, text recognition is performed on the first image to obtain the text recognition result, and the corresponding image text positioning box is transformed into the text plane coordinate system to obtain the tracking target positioning box. Subsequent images can be tracked based on this tracking target positioning box, effectively reducing the frequency of text recognition and reducing resource consumption.

[0202] Figure 9 A schematic diagram of a display control device based on an augmented reality device according to an embodiment of this application is provided. (Reference) Figure 9 The augmented reality device provided in this application is equipped with a camera and a screen. The display control device based on the augmented reality device includes an image acquisition module 31, an image conversion module 32, a target tracking module 33, a position determination module 34, and a display control module 35.

[0203] The system includes: an image acquisition module 31 for acquiring a first image captured by a camera on a text plane; an image conversion module 32 for converting a second image into a tracking target image based on a preset homography matrix and the target image size, wherein the target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system; the second image being either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition; and a homography matrix recording a mapping relationship for converting the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system; a target tracking module 33 for performing feature point matching between the tracking target image and the first image to obtain a target transformation matrix, which records a mapping relationship for converting the tracking target image from the text plane coordinate system to the camera coordinate system; a position determination module 34 for determining the target tracking position of the tracking target image in the camera coordinate system based on the target transformation matrix; and a display control module 35 for displaying a preset image on the screen based on the target tracking position.

[0204] The above describes a process where a first image is captured by a camera on a text plane. A second image is then transformed based on a preset homography matrix and the target image size to obtain a tracking target image. Feature point matching is performed between the tracking target image and the first image to obtain a target transformation matrix. The target tracking position in the camera coordinate system is determined based on this matrix, and a preset image is displayed on the screen according to this target tracking position. The target image size is determined based on the size of the text recognition result of the second image in the text plane coordinate system. The second image is either the first image or an image captured by an augmented reality device before the first image and subjected to text recognition. The homography matrix can be used to transform the text recognition result into a target image. The corresponding text image is transformed from the camera coordinate system to the text plane coordinate system, so that the tracking target image can correctly reflect the result of transforming the text recognition result of the second image to the text plane coordinate system. The target transformation matrix obtained by matching the feature points of the tracking target image and the first image can accurately transform the tracking target image to the camera coordinate system, and can accurately determine the target tracking position corresponding to the tracking target image in the camera coordinate system. When the user wears the augmented reality device, the image displayed on the screen observed by the user can be correctly superimposed on the real text object corresponding to the text recognition result, so that the information related to the text recognition result can be correctly superimposed on the real text object for display, further improving the user experience of wearing the augmented reality device.

[0205] In one possible embodiment, the target image size includes the target image width and the target image height. The display control device based on the augmented reality device further includes a size determination module, which is used for:

[0206] The width of the target image is determined based on the first side length of the image text positioning box, and the image text positioning box is determined based on the text recognition result of the second image;

[0207] The height of the target image is determined by multiplying the ratio of the height to the width of the tracking target bounding box with the width of the target image. The tracking target bounding box is obtained by transforming the image text bounding box to the text plane coordinate system.

[0208] In one possible embodiment, the display control device based on the augmented reality device further includes a matrix determination module, which is used for:

[0209] The homography matrix is ​​determined based on the tracking target location box and the camera's intrinsic parameter matrix corresponding to the text recognition result of the second image in the text plane coordinate system, as well as the preset unit conversion ratio between image distance units and spatial distance units. Image distance units are used to represent distance units on the image, and spatial distance units are used to represent distance units in space.

[0210] In one possible embodiment, the position determination module 34 determines the target tracking position corresponding to the tracking target image in the camera coordinate system according to the target transformation matrix, including: transforming the tracking target positioning box corresponding to the text recognition result of the second image in the text plane coordinate system to the camera coordinate system according to the target transformation matrix, so as to obtain the target tracking position corresponding to the tracking target image in the camera coordinate system.

[0211] In one possible embodiment, the display control module 35 displays a preset image on the screen according to the target tracking position, including:

[0212] Based on the intrinsic and extrinsic parameters of the rendering camera, the target tracking position is transformed from the camera coordinate system to the rendering camera coordinate system to obtain the rendering position information;

[0213] Display a preset image on the screen based on the rendering location information.

[0214] In one possible embodiment, the display control device based on the augmented reality device further includes a recognition response module, which is used for:

[0215] Under the condition that the text recognition triggering condition is met, the text recognition result is obtained by performing text recognition on the first image;

[0216] Determine the target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system.

[0217] In one possible embodiment, the recognition response module determines the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system, including:

[0218] Determine the first normal vector of the text plane corresponding to the text recognition result and the first distance from the text plane to the camera;

[0219] Based on the image text localization box corresponding to the text recognition result, the camera's intrinsic parameter matrix, the first normal vector, and the first distance, determine the recognized text localization box in the camera coordinate system.

[0220] Determine the text plane coordinate system based on the identified text positioning box;

[0221] Determine the target location box corresponding to the identified text location box in the text plane coordinate system.

[0222] In one possible embodiment, the recognition response module determines the recognized text positioning box in the camera coordinate system based on the image text positioning box corresponding to the text recognition result, the camera's intrinsic parameter matrix, the first normal vector, and the first distance, including:

[0223] Based on the vertex coordinates of the image text localization box corresponding to the text recognition result and the camera's intrinsic parameter matrix, determine the unit direction vector from the origin of the camera coordinate system to the recognized text localization box.

[0224] Based on the unit direction vector, the first normal vector, and the first distance, determine the corresponding recognition text bounding box in the camera coordinate system.

[0225] In one possible embodiment, the recognition response module determines the text plane coordinate system based on the recognized text positioning box, including:

[0226] The origin of the text plane coordinate system is determined based on the first vertex of the identified text positioning box;

[0227] The third direction axis of the text plane coordinate system is determined by the negative value of the first normal vector of the text plane;

[0228] The first direction axis of the text plane coordinate system is determined based on the mean of the direction vectors of the first and third sides relative to the identified text positioning box.

[0229] The second direction axis of the text plane coordinate system is determined by the cross product of the third direction axis and the first direction axis;

[0230] Define the text plane coordinate system formed by the origin, the first direction axis, the second direction axis, and the third direction axis.

[0231] In one possible embodiment, the recognition response module determines the tracking target location box corresponding to the recognized text location box in the text plane coordinate system, including:

[0232] The first vertex of the target location box is determined based on the origin of the text plane coordinate system;

[0233] The width of the tracking target positioning box is determined based on the average projection length of the first and third sides of the identified text positioning box on the first direction axis of the text plane coordinate system.

[0234] The height of the tracking target bounding box is determined based on the average projection length of the second and fourth sides of the identified text bounding box onto the second direction axis of the text plane coordinate system.

[0235] It is worth noting that in the above embodiments of the display control device based on augmented reality devices, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0236] This application also provides a display control device based on an augmented reality device, which can integrate the display control apparatus based on an augmented reality device provided in this application. Figure 10 This is a schematic diagram of a display control device based on an augmented reality device provided in an embodiment of this application. (Reference) Figure 10 The augmented reality-based display control device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when one or more programs are executed by one or more processors 41, the one or more processors 41 implement the augmented reality-based display control method provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0237] The memory 42, as a computing device-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the display control method based on an augmented reality device provided in any embodiment of this application (e.g., the image acquisition module 31, image conversion module 32, target tracking module 33, position determination module 34, and display control module 35 in the display control device based on an augmented reality device). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely located relative to the processor 41, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0238] Input device 43 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 44 may include display devices such as a display screen.

[0239] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned display control method based on augmented reality device.

[0240] The display control device, apparatus, and computer based on augmented reality devices provided above can be used to execute the display control method based on augmented reality devices provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0241] This application embodiment also provides a storage medium for storing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute the display control method based on an augmented reality device provided in the above embodiment. The display control method based on an augmented reality device includes: acquiring a first image obtained by a camera capturing an image of a text plane; transforming a second image to obtain a tracking target image according to a preset homography matrix and a target image size, wherein the target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system, the second image being either the first image or an image captured by the augmented reality device before the first image and subjected to text recognition, the homography matrix recording a mapping relationship for transforming the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system; performing feature point matching on the tracking target image and the first image to obtain a target transformation matrix, the target transformation matrix recording a mapping relationship for transforming the tracking target image from the text plane coordinate system to the camera coordinate system; determining the target tracking position of the tracking target image in the camera coordinate system according to the target transformation matrix; and displaying a preset image on the screen according to the target tracking position.

[0242] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0243] Of course, the computer-executable instructions stored in the storage medium provided in the embodiments of this application are not limited to the display control method based on augmented reality devices provided above, but can also execute related operations in the display control method based on augmented reality devices provided in any embodiment of this application.

[0244] The display control device, equipment, and storage medium based on augmented reality devices provided in the above embodiments can execute the display control method based on augmented reality devices provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the display control method based on augmented reality devices provided in any embodiment of this application.

[0245] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments provided herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A display control method based on an augmented reality device, characterized in that, The augmented reality device is equipped with a camera and a screen, and the display control method includes: Acquire a first image obtained by the camera capturing an image of the text plane; Based on the preset homography matrix and the target image size, the second image is transformed to obtain the tracking target image. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is the first image or an image captured by the augmented reality device before the first image and subjected to text recognition. The homography matrix records the mapping relationship used to transform the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system. Feature point matching is performed on the tracking target image and the first image to obtain a target transformation matrix. The target transformation matrix records the mapping relationship used to transform the tracking target image from the text plane coordinate system to the camera coordinate system. The target tracking position of the target image in the camera coordinate system is determined based on the target transformation matrix. A preset image is displayed on the screen based on the target tracking position.

2. The display control method based on augmented reality devices according to claim 1, characterized in that, The target image size includes the target image width and the target image height. Before converting the second image according to the preset homography matrix and the target image size to obtain the tracking target image, the method further includes: The width of the target image is determined based on the first side length of the image text positioning box, and the image text positioning box is determined based on the text recognition result of the second image; The height of the target image is determined by multiplying the ratio of the height to the width of the tracking target positioning box with the width of the target image. The tracking target positioning box is obtained by transforming the image text positioning box to the text plane coordinate system.

3. The display control method based on augmented reality devices according to claim 1, characterized in that, Before converting the second image according to the preset homography matrix and the target image size to obtain the tracking target image, the method further includes: The homography matrix is ​​determined based on the tracking target positioning box corresponding to the text recognition result of the second image in the text plane coordinate system, the intrinsic parameter matrix of the camera, and the preset unit conversion ratio between image distance units and spatial distance units. The image distance units are used to represent distance units on the image, and the spatial distance units are used to represent distance units in space.

4. The display control method based on augmented reality devices according to claim 1, characterized in that, Determining the target tracking position of the target image in the camera coordinate system based on the target transformation matrix includes: Based on the target transformation matrix, the text recognition result of the second image is transformed into the tracking target location box corresponding to the text plane coordinate system to the camera coordinate system, so as to obtain the target tracking position of the tracking target image corresponding to the camera coordinate system.

5. The display control method based on augmented reality devices according to claim 1, characterized in that, The step of displaying a preset image on the screen according to the target tracking position includes: Based on the intrinsic and extrinsic parameters of the rendering camera, the target tracking position is transformed from the camera coordinate system to the rendering camera coordinate system to obtain the rendering position information; A preset image is displayed on the screen based on the rendering position information.

6. The display control method based on augmented reality devices according to claim 1, characterized in that, After acquiring the first image obtained by the camera capturing an image of the text plane, the method further includes: If the text recognition triggering conditions are met, text recognition is performed on the first image to obtain the text recognition result; Determine the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system.

7. The display control method based on augmented reality devices according to claim 6, characterized in that, Determining the tracking target location box corresponding to the image text location box of the text recognition result in the text plane coordinate system includes: Determine the first normal vector of the text plane corresponding to the text recognition result and the first distance from the text plane to the camera; Based on the image text localization box corresponding to the text recognition result, the intrinsic parameter matrix of the camera, the first normal vector, and the first distance, the recognition text localization box corresponding to the image text localization box in the camera coordinate system is determined; The text plane coordinate system is determined based on the identified text positioning box; Determine the tracking target positioning box corresponding to the identified text positioning box in the text plane coordinate system.

8. The display control method based on augmented reality devices according to claim 7, characterized in that, The step of determining the recognized text location box in the camera coordinate system based on the image text location box corresponding to the text recognition result, the intrinsic parameter matrix of the camera, the first normal vector, and the first distance includes: Based on the vertex coordinates of the image text location box corresponding to the text recognition result and the intrinsic parameter matrix of the camera, determine the unit direction vector from the origin of the camera coordinate system to the recognized text location box. Based on the unit direction vector, the first normal vector, and the first distance, the corresponding recognition text positioning box in the camera coordinate system is determined.

9. The display control method based on augmented reality devices according to claim 7, characterized in that, Determining the text plane coordinate system based on the identified text positioning box includes: The origin of the text plane coordinate system is determined based on the first vertex of the identified text positioning box; The third direction axis of the text plane coordinate system is determined by the negative value of the first normal vector of the text plane; The first direction axis of the text plane coordinate system is determined based on the mean of the direction vectors of the first and third sides of the identified text positioning box. The second direction axis of the text plane coordinate system is determined based on the cross product of the third direction axis and the first direction axis; The text plane coordinate system is defined by the origin, the first direction axis, the second direction axis, and the third direction axis.

10. The display control method based on an augmented reality device according to claim 7, characterized in that, Determining the tracking target location box corresponding to the identified text location box in the text plane coordinate system includes: The first vertex of the target location box is determined based on the origin of the text plane coordinate system; The width of the tracking target positioning box is determined based on the average projection length of the first and third sides of the identified text positioning box on the first direction axis of the text plane coordinate system. The height of the tracking target positioning box is determined based on the average projection length of the second and fourth sides of the identified text positioning box on the second direction axis of the text plane coordinate system.

11. A display control device based on an augmented reality device, characterized in that, The augmented reality device is equipped with a camera and a screen, and the display control device includes an image acquisition module, an image conversion module, a target tracking module, a position determination module, and a display control module, wherein: The image acquisition module is used to acquire a first image obtained by the camera capturing an image of the text plane; The image conversion module is used to convert the second image into a tracking target image according to a preset homography matrix and the target image size. The target image size is determined based on the size corresponding to the text recognition result of the second image in the text plane coordinate system. The second image is the first image or an image captured by the augmented reality device before the first image and subjected to text recognition. The homography matrix records the mapping relationship used to convert the text image corresponding to the text recognition result from the camera coordinate system to the text plane coordinate system. The target tracking module is used to perform feature point matching between the tracked target image and the first image to obtain a target transformation matrix. The target transformation matrix records a mapping relationship for transforming the tracked target image from the text plane coordinate system to the camera coordinate system. The position determination module is used to determine the target tracking position of the tracked target image in the camera coordinate system according to the target transformation matrix; The display control module is used to display a preset image on the screen according to the target tracking position.

12. A display control device based on an augmented reality device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the display control method based on an augmented reality device as described in any one of claims 1-10.

13. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the display control method based on an augmented reality device as described in any one of claims 1-10.

Images