A transparent display interaction method for augmented reality

Abstract

The present invention relates to the field of image processing technology, and in particular to a transparent display interaction method for augmented reality, including: obtaining corresponding touch data, coordinates of the user's touch center on the display screen, and real-time image data, and then obtaining the user's touch coordinates in the real image, that is, obtaining the scene represented by the screen touch point; then obtaining the optimal boundary of the recognized image with the touch point of the real image, and generating corresponding virtual information; finally determining the real-time display position of the virtual image. By determining the real-time display position of the virtual image, the present invention can improve the recognition of the user's touch interaction intention, display more accurate virtual information, help the system understand the user's touch operation intention, and improve the accuracy of virtual information display.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a transparent display interaction method for augmented reality. Background Art

[0002] Transparent display interaction of augmented reality is a technology that combines virtual digital information with the real world. It enables users to perceive virtual information superimposed on the real world while seeing the real world through specific display technology and interaction methods. Transparent display of intelligent driving has shown a good development trend in recent years. The combination of virtual information and actual road conditions is integrated into the front windshield of intelligent driving, so as to provide a seamless and integrated driving experience. A common transparent display method is to use a specially treated transparent display screen as a windshield, which can display virtual information and allow real light to pass through. Through touch and other interactive methods, the display of virtual information can be realized, thereby achieving the purpose of augmented reality. However, since real light can pass through the screen, in touch interaction, different observation angles of users will cause different scenes represented by the touch point. If the user's touch point is judged to represent the scene at a fixed perspective, the error rate of virtual information obtained by touch interaction will be high; and intelligent driving is usually not static, and fixed display of virtual information will cause image tearing and dislocation. Reduce the accuracy of virtual information display. Summary of the invention

[0003] The present invention provides a transparent display interaction method for augmented reality to solve the existing problems.

[0004] A transparent display interaction method for augmented reality of the present invention adopts the following technical solution:

[0005] An embodiment of the present invention provides a transparent display interaction method for augmented reality, the method comprising the following steps:

[0006] Obtain the distance between the user's eyes and the camera, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen;

[0007] Perform feature detection on the images corresponding to the three cameras at each moment to obtain a number of identical features; the features are composed of a number of pixels; according to the coordinates of each pixel of each feature and the number of pixels contained in each feature, obtain the pixel distance difference of each feature in the images corresponding to any two cameras at each moment;

[0008] Obtain a real image after the images corresponding to all cameras at each moment are fused; obtain the horizontal and vertical coordinates of the touch point in the real image at the current moment according to the pixel distance difference of each feature in the images corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinates of the user's touch center on the display screen, and the distance between the user's eyes and the camera;

[0009] According to the gray value of each pixel in the real image at the current moment, the virtual image at the current moment is obtained;

[0010] Taking the horizontal coordinate and vertical coordinate of the touch point in the real image at the current moment as the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment, and obtaining the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the next moment in the future according to the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment;

[0011] According to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, the augmented reality function of the virtual image at the next moment in the future is completed.

[0012] Furthermore, the pixel distance difference of each feature in the images corresponding to any two cameras at each moment is obtained according to the coordinates of each pixel point of each feature and the number of pixels contained in each feature, and the specific steps include the following:

[0013] Construct the coordinate system of the image corresponding to each camera at each moment, and obtain the coordinates of each pixel in the image corresponding to each camera;

[0014] According to the horizontal coordinate of each pixel point of each feature and the number of pixels contained in each feature, the pixel distance difference of each feature in the image corresponding to the first and second cameras at the current moment is obtained;

[0015] According to the ordinate of each pixel point of each feature and the number of pixels contained in each feature, the pixel distance difference of each feature in the images corresponding to the second and third cameras at the current moment is obtained.

[0016] Furthermore, the specific calculation formula for obtaining the pixel distance difference of each feature in the image corresponding to the first and second cameras at the current moment according to the horizontal coordinate of each pixel point of each feature and the number of pixels contained in each feature is:

[0017]

[0018] Among them, l 1,2,i Represents the pixel distance difference between the i-th feature in the image corresponding to the first and second cameras at the current moment, N iIndicates the number of pixels contained in the i-th feature; qx 1,i,j Represents the horizontal coordinate of the jth pixel of the i-th feature in the image corresponding to the first camera at the current moment; qx 2,i,j Represents the horizontal coordinate of the j-th pixel of the ith feature in the image corresponding to the second camera at the current moment.

[0019] Furthermore, the specific calculation formula for obtaining the pixel distance difference of each feature in the image corresponding to the second and third cameras at the current moment according to the ordinate of each pixel point of each feature and the number of pixels contained in each feature is:

[0020]

[0021] Among them, l 2,3,i Represents the pixel distance difference between the i-th feature in the image corresponding to the second and third cameras at the current moment, N i Indicates the number of pixels contained in the i-th feature; qy 2,i,j Represents the ordinate of the jth pixel of the ith feature in the image corresponding to the second camera at the current moment; qy 3,i,j Represents the ordinate of the j-th pixel of the ith feature in the image corresponding to the third camera at the current moment.

[0022] Furthermore, the specific calculation formula for obtaining the horizontal coordinate and vertical coordinate of the touch point in the real image at the current moment according to the pixel distance difference of each feature in the image corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinate of the user's touch center on the display screen, and the distance between the user's eyes and the camera is:

[0023]

[0024] Among them, x s Represents the horizontal coordinate of the touch point in the real image at the current moment, y s represents the ordinate of the touch point in the real image at the current moment, x0 represents the abscissa of the touch center of the user on the display screen at the current moment, y0 represents the ordinate of the touch center of the user on the display screen at the current moment; L1 represents the distance between the first camera and the second camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the first and second cameras at the current moment; x ′ represents the average of the horizontal distances between the user's eyes and the three cameras; L2 represents the distance between the second camera and the third camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the second and third cameras at the current moment; y ′Represents the average of the vertical distances between the user's eyes and the three cameras, where the distance between the user's eyes and the camera consists of the horizontal distance and the vertical distance.

[0025] Furthermore, the step of obtaining the virtual image at the current moment according to the gray value of each pixel in the real image at the current moment includes the following specific steps:

[0026] According to the gray value of each pixel in the real image at the current moment, a final extended window of the touch point in the real image at the current moment is obtained;

[0027] Record the boundary of the minimum circumscribed rectangle of the final extended window in the real image at the current moment as the optimal boundary of the touch point in the real image at the current moment;

[0028] A virtual image at the current moment is generated according to an image within an optimal boundary of the touch point in the real image at the current moment.

[0029] Furthermore, the method of obtaining the final extended window of the touch point in the real image at the current moment according to the gray value of each pixel in the real image at the current moment includes the following specific steps:

[0030] In the real image at the current moment, according to the preset window size, the initial window center is set as the coordinate of the touch point, and the window is increased cyclically in four directions in turn, up, down, left and right. The size of each expansion is the preset window size. When the absolute value of the difference between the grayscale mean of all pixels in the extended window and the grayscale mean of all pixels in the real image is greater than the absolute value of the difference between the grayscale mean of all pixels in the initial window and the grayscale mean of all pixels in the real image, the window expansion ends, and the final expanded window of the touch point in the real image at the current moment is obtained.

[0031] Furthermore, the step of obtaining the abscissa and ordinate of the dynamic display point of the virtual image at the next moment in the future according to the abscissa and ordinate of the dynamic display point of the virtual image at the current moment includes the following specific steps:

[0032] Record the real images at the previous H moments adjacent to the real image at the current moment as reference images of the real image at the current moment, where H is a preset number threshold;

[0033] The images corresponding to the three cameras at each moment and the corresponding real images are subjected to feature detection using a scale-invariant feature transformation algorithm to obtain the features of the real images at each moment;

[0034] The pixel point corresponding to the mean of the horizontal and vertical coordinates of the central pixel points of all features in each reference image corresponding to the real image at the current moment is recorded as the key point of each reference image;

[0035] Perform curve fitting on the key points in all reference images corresponding to the real image at the current moment, and obtain the fitting curve of the key points in all reference images corresponding to the real image at the current moment;

[0036] According to the coordinates of the dynamic display points of the virtual image at the current moment, the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment, and the tangent slope of the key points on the fitting curve, the horizontal and vertical coordinates of the dynamic display points of the virtual image at the next moment in the future are obtained.

[0037] Furthermore, the specific calculation formula for obtaining the abscissa and ordinate of the dynamic display point of the virtual image at the next moment in the future according to the coordinates of the dynamic display point of the virtual image at the current moment, the abscissa of the key points in all reference images corresponding to the real image at the current moment, and the tangent slope of the key points on the fitting curve is:

[0038] dx s+1 =x s +Δx s

[0039] dy s+1 =y s +k s ×Δx s

[0040] Among them, dx s+1 The horizontal coordinate of the dynamic display point of the virtual image at the next moment in the future, dy s+1 Indicates the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, x s Indicates the horizontal coordinate of the dynamic display point of the virtual image at the current moment, y s The vertical coordinate of the dynamic display point of the virtual image at the current moment, Δx s k represents the average rate of change of the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment; s Represents the mean of the tangent slopes of the key points on the fitting curve in all reference images corresponding to the real image at the current moment.

[0041] Furthermore, the method of completing the augmented reality function of the virtual image at the next moment in the future according to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future includes the following specific steps:

[0042] The horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment are updated to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, so as to obtain the display position of the virtual image at the next moment in the future;

[0043] According to the display position of the virtual image at the next moment in the future, the virtual image at the next moment in the future is displayed on the display screen, completing the augmented reality function of the virtual image at the next moment in the future.

[0044] The beneficial effects of the technical solution of the present invention are:

[0045] In the embodiment of the present invention, the distance between the user's eyes and the camera, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen are obtained; feature detection is performed on the images corresponding to the three cameras at each moment to obtain a number of identical features; the features are composed of a number of pixels; according to the coordinates of each pixel of each feature and the number of pixels contained in each feature, the pixel distance difference of each feature in the image corresponding to any two cameras at each moment is obtained; according to the coordinates of the user touching the display screen, and the analysis of the image captured by the camera and the distance information of the user's eyes, the system can more accurately identify and understand the user's intention, improve the accuracy and response speed of the interaction. The real image after the images corresponding to all cameras at each moment are obtained; according to the pixel distance difference of each feature in the image corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinates of the user's touch center on the display screen, and the distance between the user's eyes and the camera, the horizontal and vertical coordinates of the touch point in the real image at the current moment are obtained; according to the analysis of the features in the images collected by multiple cameras, combined with the geometric relationship between the cameras and the real distance between the user and the camera, the positioning accuracy of the touch point is significantly improved. According to the gray value of each pixel in the real image at the current moment, the virtual image at the current moment is obtained; according to the analysis of the touch points in the real image at the current moment, the optimal boundary is determined to obtain the virtual image at the current moment, further improving the accuracy of virtual information display. The horizontal coordinates and vertical coordinates of the touch points in the real image at the current moment are used as the horizontal coordinates and vertical coordinates of the dynamic display points of the virtual image at the current moment, and the horizontal coordinates and vertical coordinates of the dynamic display points of the virtual image at the next moment in the future are obtained according to the horizontal coordinates and vertical coordinates of the dynamic display points of the virtual image at the current moment. According to the analysis of the images of the real image at several moments before the current moment, a more accurate dynamic display point position can be calculated. It is helpful to simulate and display the dynamic changes of the touch points more accurately in the virtual environment and improve the user experience. According to the horizontal coordinates and vertical coordinates of the dynamic display points of the virtual image at the next moment in the future, the augmented reality function of the virtual image at the next moment in the future is completed. So far, the present invention can improve the recognition of the user's touch interaction intention by determining the real-time display position of the virtual image, display more accurate virtual information, help the system understand the user's touch operation intention, and improve the accuracy of virtual information display. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a transparent display interaction method for augmented reality according to the present invention;

[0048] Figure 2 This is a schematic diagram of camera installation;

[0049] Figure 3 A schematic diagram of window expansion. DETAILED DESCRIPTION

[0050] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the transparent display interaction method for augmented reality proposed by the present invention, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0051] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0052] A specific scheme of a transparent display interaction method for augmented reality provided by the present invention is described in detail below with reference to the accompanying drawings.

[0053] See also Figure 1 , which shows a flowchart of a transparent display interaction method for augmented reality provided by an embodiment of the present invention, the method comprising the following steps:

[0054] Step S001: Obtain the distance between the user's eyes and the camera, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen.

[0055] The front windshield of the car is used as a transparent display device, and three cameras are placed in a position in the middle of the car that will not be blocked (such as the rearview mirror) to estimate the touch coordinate deviation of the real-time scene caused by the difference in the user's observation angle.

[0056] A rectangular coordinate system is constructed with the upper left corner of the display screen as the origin, the horizontal rightward as the horizontal axis, and the horizontal downward as the vertical axis.

[0057] Through vehicle integration, the distance between the user's eyes and the three cameras, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen are collected.

[0058] What needs to be explained is: Figure 2 Schematic diagram of camera installation in this embodiment, wherein S1, S2 and S3 are three cameras respectively, L1 is the distance between the first camera and the second camera, L2 is the distance between the second camera and the third camera, S1 and S2 are located on the same horizontal line, S3 and S2 are located on the same vertical line, and the image corresponding to each camera is of the same size, wherein the distance between the user's eye and the camera is composed of the horizontal distance and the vertical distance.

[0059] Step S002: Perform feature detection on the images corresponding to the three cameras at each moment to obtain a number of identical features; the features are composed of a number of pixel points; based on the coordinates of each pixel point of each feature and the number of pixel points contained in each feature, obtain the pixel distance difference of each feature in the images corresponding to any two cameras at each moment.

[0060] The front windshield of a car is an excellent location for a display screen. However, the scene observed through the display screen is generally different depending on the user's position in the car. Therefore, when a user performs a touch operation, even if the touch point is in the same position, the actual objects that need to be operated are different due to different observation angles. If the user specifies a coordinate of the transparent device and observes it in a fixed direction at the same time, the scene represented by the coordinate will also shift to the same side. For example, when the user observes from the center position to the left, the overall scene shifts to the left, so the coordinates of the object that the user clicks on will also shift to the left. Therefore, when a user clicks on a touch coordinate, it is necessary to analyze the coordinate position of the click in the real image.

[0061] When observing on the same horizontal line, the scene changes along the horizontal line. Correspondingly, when observing along the same vertical line, the scene changes along the vertical line, and the horizontal change distance of the observation angle is linearly positively correlated with the pixel distance of the characteristic horizontal change of the image.

[0062] The same-sized images acquired by the three cameras at the current moment are subjected to feature detection using a scale-invariant feature transformation algorithm, and the same features are marked.

[0063] Overlap two images at the same level to obtain the pixel change distance of the same feature in the horizontal direction. The average of the pixel change distance is recorded as the pixel difference between the two images. The horizontal distance of the user's eyes relative to the center position is obtained, and the horizontal offset of the user's actual observation can be obtained, and then the coordinates of the scene that the user actually wants to touch can be obtained.

[0064] With the upper left corner of the image as the origin, the horizontal rightward as the horizontal axis, and the horizontal downward as the vertical axis, a coordinate system of the image corresponding to each camera at each moment is constructed to obtain the coordinates of each pixel in the image corresponding to each camera.

[0065] Taking the i-th feature as an example, the calculation formula for the pixel distance difference of the i-th feature in the images corresponding to the first and second cameras at the current moment is:

[0066]

[0067] Among them, l 1,2,i Represents the pixel distance difference between the i-th feature in the image corresponding to the first and second cameras at the current moment, N i Indicates the number of pixels contained in the i-th feature; qx 1,i,j Represents the horizontal coordinate of the jth pixel of the i-th feature in the image corresponding to the first camera at the current moment; qx 2,i,j Represents the horizontal coordinate of the j-th pixel of the ith feature in the image corresponding to the second camera at the current moment.

[0068] What needs to be explained is: qx 2,i,j Represents the horizontal coordinate of the jth pixel of the ith feature in the image corresponding to the second camera at the current moment; since the origin of the coordinate system of the image corresponding to each camera is located in the upper left corner, and the first camera is located further to the left, then the ith feature in the image corresponding to the first camera at the current moment is further to the left than the ith feature in the image corresponding to the second camera, and has a smaller horizontal coordinate. Among them, the first and second cameras are Figure 2 S1 and S2 in .

[0069] According to the above method, this embodiment can obtain the pixel distance difference of all features in the image corresponding to the first camera and the second camera at the current moment.

[0070] Taking the i-th feature as an example, the calculation formula for the pixel distance difference of the i-th feature in the images corresponding to the second and third cameras at the current moment is:

[0071]

[0072] Among them, l 2,3,iRepresents the pixel distance difference between the i-th feature in the image corresponding to the second and third cameras at the current moment, N i Indicates the number of pixels contained in the i-th feature; qy 2,i,j Represents the ordinate of the jth pixel of the ith feature in the image corresponding to the second camera at the current moment; qy 3,i,j Represents the ordinate of the j-th pixel of the ith feature in the image corresponding to the third camera at the current moment.

[0073] What needs to be explained is: 3,i,j Indicates the ordinate of the jth pixel of the ith feature in the image corresponding to the third camera at the current moment; since the origin of the coordinate system of the image corresponding to each camera is located in the upper left corner, and the second camera is located lower, the ith feature in the image corresponding to the second camera is lower than the ith feature in the image corresponding to the third camera, and has a larger ordinate. Among them, the second and third cameras are Figure 2 S2 and S3 in .

[0074] According to the above method, this embodiment can obtain the pixel distance differences of all features in the images corresponding to the second and third cameras at the current moment.

[0075] Step S003: Obtain a real image after the images corresponding to all cameras at each moment are fused; obtain the horizontal and vertical coordinates of the touch point in the real image at the current moment according to the pixel distance difference of each feature in the images corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinates of the user's touch center on the display screen, and the distance between the user's eyes and the camera.

[0076] The image registration technology is used to perform feature fusion on the images corresponding to the three cameras at each moment, and a real image is obtained after the images corresponding to all cameras at each moment are fused.

[0077] It should be noted that the size of the real image is consistent with the number of pixels on the display screen. The scale-invariant feature transformation algorithm and image registration technology are both well-known technologies, and the specific methods are not introduced here.

[0078] The images corresponding to the three cameras simulate the differences in the scenes observed by a person under horizontal and vertical changes, thereby obtaining the coordinates of the touch point in the real image at the current moment.

[0079] Taking the real image at the current moment as an example, the calculation formulas for the horizontal and vertical coordinates of the touch point in the real image at the current moment are:

[0080]

[0081] Among them, xs Represents the horizontal coordinate of the touch point in the real image at the current moment, y s represents the ordinate of the touch point in the real image at the current moment, x0 represents the abscissa of the touch center of the user on the display screen at the current moment, y0 represents the ordinate of the touch center of the user on the display screen at the current moment; L1 represents the distance between the first camera and the second camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the first and second cameras at the current moment; x ′ represents the average of the horizontal distances between the user's eyes and the three cameras; L2 represents the distance between the second camera and the third camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the second and third cameras at the current moment; y ′ Indicates the average vertical distance between the user's eyes and the three cameras.

[0082] It should be noted that: x0 represents the horizontal coordinate of the user's touch center on the display screen at the current moment, and y0 represents the vertical coordinate of the user's touch center on the display screen at the current moment. It usually remains unchanged during the touch and press process. It is affected by different observation angles of the user, and the scene represented by the touch point is also different. Since the distance between the three cameras and the display screen and the distance between the user and the display screen are relatively small, it is considered that the user and the camera are in the same plane.

[0083] According to the above method, this embodiment can obtain the horizontal coordinate and the vertical coordinate of the touch point in the real image at the current moment.

[0084] Step S004: Obtain the virtual image at the current moment according to the gray value of each pixel in the real image at the current moment.

[0085] In order to analyze the user's touch intention, it is necessary to analyze the pixels around the touch point and identify the content of the image. The basic method is to determine the boundary of the identified content and cut it based on the boundary.

[0086] It should be noted that: the trained YOLO target detection algorithm is usually used to obtain the boundary of the touch point in the real image at the current moment, wherein the YOLO target detection algorithm is a well-known technology. The main neural network model used by the YOLO target detection algorithm in this embodiment is a convolutional neural network, and the data set used is a real image data set.

[0087] The pixels that need to be segmented are divided into two categories, that is, the labeling process of the training set is: single-channel semantic label, the corresponding position pixel belongs to the background area is marked as 0, and the boundary of the touch point is marked as 1. The loss function used is the cross entropy loss function. Among them, the input is a real image containing touch points, and the output is the boundary of the touch point in the real image. The specific training process is well known and will not be described in detail in this embodiment.

[0088] However, the choice of boundary usually affects the efficiency of recognition. If the boundary is too large, too much content will be recognized, resulting in unclear analysis of the user's intention; if the boundary is too small, the user's intention cannot be recognized. Therefore, it is necessary to calculate the optimal boundary of the touch point.

[0089] Generally, for scenes displayed by user touch, such as buildings, there is a large grayscale difference between the target scene and the background, while the grayscale difference of the target scene pixels is small.

[0090] In the real image at the current moment, according to the preset window size, let the initial window center be the coordinate of the touch point, and cyclically increase the window in four directions, up, down, left, and right, respectively. The size of each expansion is the preset window size, such as Figure 3 As shown, when the absolute value of the difference between the grayscale mean of all pixels in the extended window and the grayscale mean of all pixels in the real image is greater than the absolute value of the difference between the grayscale mean of all pixels in the initial window and the grayscale mean of all pixels in the real image, the window expansion ends and the final extended window of the touch point in the real image at the current moment is obtained.

[0091] It should be noted that: the preset window size in this embodiment is 3×3, and this is taken as an example for description. Figure 3 is a schematic diagram of window expansion in this embodiment, according to Figure 3 As shown, the initial window is expanded for 5 times. The initial window first expands to the right by a preset window size, then expands to the top, left and bottom by a preset window size respectively, and finally continues to expand to the right by a preset window size.

[0092] The boundary of the minimum circumscribed rectangle of the final extended window in the real image at the current moment is recorded as the optimal boundary of the touch point in the real image at the current moment.

[0093] The image within the optimal boundary is input into the trained YOLO target detection algorithm to obtain a list of scene information clicked by the user, and then the obtained scene information list is input into a CNN-based text mining network to obtain manually annotated intent information, and finally the user's intent information is obtained; the user's intent information is processed, and then the data is displayed. The display method can adopt a template display, and the relevant processed information is input into the template image, thereby generating a virtual image at the current moment.

[0094] It should be noted that the YOLO target detection algorithm and the CNN-based text mining network are both well-known technologies, and the specific methods are not introduced here.

[0095] Step S005: Use the horizontal coordinate and vertical coordinate of the touch point in the real image at the current moment as the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment, and obtain the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the next future moment according to the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment.

[0096] During the user's touch process, due to reasons such as vehicle operation, the scenery in the image will be in motion. If the virtual image is still displayed based on the touch point in the real image at the current moment, the displayed virtual image may be seriously misaligned because the scenery has moved, causing a tearing sensation and affecting the user experience.

[0097] Therefore, it is necessary to estimate the direction of scene change, so as to determine the dynamic position display of the virtual image. In adjacent moments, it is assumed that the scene changes at a uniform speed in the image. The feature points in the real image at adjacent moments are extracted by existing methods, and the change trend is calculated. Combined with the touch point of the real image at the current moment, the coordinates of the dynamic display point of the virtual image at the next moment in the future are estimated.

[0098] The real images at the previous H moments adjacent to the real image at the current moment are recorded as reference images of the real image at the current moment. The images corresponding to the three cameras at each moment and the corresponding real images are subjected to feature detection using a scale-invariant feature transformation algorithm to obtain the features of the real image at each moment.

[0099] The pixel point corresponding to the mean of the horizontal and vertical coordinates of the central pixel points of all features in each reference image corresponding to the real image at the current moment is recorded as the key point of each reference image.

[0100] The least square method is used to perform curve fitting on the key points in all reference images corresponding to the real image at the current moment, so as to obtain the fitting curve of the key points in all reference images corresponding to the real image at the current moment.

[0101] It should be noted that: in this embodiment, the preset quantity threshold H is 10, which is used as an example for description. Among them, the scale-invariant feature transformation algorithm and the least squares method are both well-known technologies, and the specific methods are not introduced here.

[0102] Since the coordinate changes of the dynamic display points of the virtual image are consistent with the changes of the key points in all reference images corresponding to the real image at the current moment, the coordinates of the dynamic display points of the virtual image at the next moment in the future can be obtained.

[0103] The horizontal coordinate and the vertical coordinate of the touch point in the real image at the current moment are used as the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment;

[0104] It should be noted that the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment are the horizontal coordinate and the vertical coordinate of the central pixel point of the virtual image at the current moment.

[0105] According to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment, the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future are obtained.

[0106] Taking the virtual image at the next moment in the future as an example, the calculation formula for the horizontal and vertical coordinates of the dynamic display point of the virtual image at the next moment in the future is:

[0107] dx s+1 =x s +Δx s

[0108] dy s+1 =y s +k s ×Δx s

[0109] Among them, dx s+1 The horizontal coordinate of the dynamic display point of the virtual image at the next moment in the future, dy s+1 Indicates the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, x s Indicates the horizontal coordinate of the dynamic display point of the virtual image at the current moment, y s The vertical coordinate of the dynamic display point of the virtual image at the current moment, Δx s k represents the average rate of change of the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment; s Represents the mean of the tangent slopes of the key points on the fitting curve in all reference images corresponding to the real image at the current moment.

[0110] What needs to be explained is: Δx s Represents the average rate of change of the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment; the average rate of change is a well-known technology used to describe the concept of the average change speed of a quantity over a period of time or a certain interval, and is usually used to analyze the changes of functions, data sequences or other quantities over time or space.

[0111] According to the above method, this embodiment can obtain the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future.

[0112] Step S006: According to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, the augmented reality function of the virtual image at the next moment in the future is completed.

[0113] Since the display screen and the real image are of the same size, the coordinates between the two can correspond one-to-one, the coordinate changes of the scene can be considered as the coordinate changes on the display screen, and the generated virtual image only occupies a small part of the display screen.

[0114] The horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment are updated to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, so as to obtain the display position of the virtual image at the next moment in the future.

[0115] According to the display position of the virtual image at the next moment in the future, the virtual image at the next moment in the future is displayed on the display screen, completing the augmented reality function of the virtual image at the next moment in the future.

[0116] What needs to be explained is: the dynamic display point of the virtual image at the next moment is calculated step by step, and the virtual image with dynamically changing display position is displayed to reduce the misalignment phenomenon between the virtual image and the real image. If the user performs an operation, the area clicked is the area of ​​the virtual image, the position movement of the virtual image is paused, the user's task is processed, and it is displayed at the original position until the user clicks to cancel the display; if the area clicked by the user is not the virtual image area, the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the next moment in the future are continuously generated according to the method of obtaining the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, and the display position of the virtual image at the future moment is obtained.

[0117] So far, the present invention is completed.

[0118] In summary, in an embodiment of the present invention, the distance between the user's eyes and the camera, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen are obtained; feature detection is performed on the images corresponding to the three cameras at each moment to obtain a number of identical features; the features are composed of a number of pixel points; according to the coordinates of each pixel point of each feature and the number of pixel points contained in each feature, the pixel distance difference of each feature in the image corresponding to any two cameras at each moment is obtained; the real image after the images corresponding to all cameras at each moment are fused is obtained; according to the pixel distance difference of each feature in the image corresponding to any two cameras at the current moment, the pixel distance difference of any two cameras at each moment is obtained; The distance between the cameras, the coordinates of the user's touch center on the display screen, and the distance between the user's eyes and the camera are used to obtain the horizontal and vertical coordinates of the touch point in the real image at the current moment; the virtual image at the current moment is obtained according to the grayscale value of each pixel in the real image at the current moment; the horizontal and vertical coordinates of the touch point in the real image at the current moment are used as the horizontal and vertical coordinates of the dynamic display point of the virtual image at the current moment, and the horizontal and vertical coordinates of the dynamic display point of the virtual image at the next moment in the future are obtained according to the horizontal and vertical coordinates of the dynamic display point of the virtual image at the current moment; the augmented reality function of the virtual image at the next moment in the future is completed according to the horizontal and vertical coordinates of the dynamic display point of the virtual image at the next moment in the future. So far, the present invention can improve the recognition of the user's touch interaction intention by determining the real-time display position of the virtual image, display more accurate virtual information, help the system understand the user's touch operation intention, and improve the accuracy of virtual information display.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A transparent display interaction method for augmented reality, characterized in that: The method comprises the following steps: Obtain the distance between the user's eyes and the camera, the distance between any two cameras, the image corresponding to each camera at each moment, and the coordinates of the user's touch center on the display screen; Perform feature detection on the images corresponding to the three cameras at each moment to obtain a number of identical features; the features are composed of a number of pixels; according to the coordinates of each pixel of each feature and the number of pixels contained in each feature, obtain the pixel distance difference of each feature in the images corresponding to any two cameras at each moment; Obtain a real image after the images corresponding to all cameras at each moment are fused; obtain the horizontal and vertical coordinates of the touch point in the real image at the current moment according to the pixel distance difference of each feature in the images corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinates of the user's touch center on the display screen, and the distance between the user's eyes and the camera; According to the gray value of each pixel in the real image at the current moment, the virtual image at the current moment is obtained; Taking the horizontal coordinate and vertical coordinate of the touch point in the real image at the current moment as the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment, and obtaining the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the next moment in the future according to the horizontal coordinate and vertical coordinate of the dynamic display point of the virtual image at the current moment; According to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, the augmented reality function of the virtual image at the next moment in the future is completed; The specific calculation formula for obtaining the horizontal coordinate and vertical coordinate of the touch point in the real image at the current moment according to the pixel distance difference of each feature in the image corresponding to any two cameras at the current moment, the distance between any two cameras, the coordinate of the user's touch center on the display screen, and the distance between the user's eyes and the camera is: Among them, x s Represents the horizontal coordinate of the touch point in the real image at the current moment, y s represents the ordinate of the touch point in the real image at the current moment, x0 represents the abscissa of the touch center of the user on the display screen at the current moment, y0 represents the ordinate of the touch center of the user on the display screen at the current moment; L1 represents the distance between the first camera and the second camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the first and second cameras at the current moment; x ′ represents the average of the horizontal distances between the user's eyes and the three cameras; L2 represents the distance between the second camera and the third camera; Represents the mean of the pixel distance differences of all features in the images corresponding to the second and third cameras at the current moment; y ′ Represents the average of the vertical distances between the user's eyes and the three cameras, where the distance between the user's eyes and the camera consists of the horizontal distance and the vertical distance.

2. A transparent display interaction method for augmented reality according to claim 1, characterized in that: The specific steps of obtaining the pixel distance difference of each feature in the images corresponding to any two cameras at each moment according to the coordinates of each pixel point of each feature and the number of pixels contained in each feature are as follows: Construct the coordinate system of the image corresponding to each camera at each moment, and obtain the coordinates of each pixel in the image corresponding to each camera; According to the horizontal coordinate of each pixel point of each feature and the number of pixels contained in each feature, the pixel distance difference of each feature in the image corresponding to the first and second cameras at the current moment is obtained; According to the ordinate of each pixel point of each feature and the number of pixels contained in each feature, the pixel distance difference of each feature in the images corresponding to the second and third cameras at the current moment is obtained.

3. A transparent display interaction method for augmented reality according to claim 2, characterized in that: The specific calculation formula for obtaining the pixel distance difference of each feature in the image corresponding to the first and second cameras at the current moment according to the horizontal coordinate of each pixel point of each feature and the number of pixels contained in each feature is: Among them, l 1,2,i Represents the pixel distance difference between the i-th feature in the image corresponding to the first and second cameras at the current moment, N i Indicates the number of pixels contained in the i-th feature; qx 1,i,j Represents the horizontal coordinate of the jth pixel of the i-th feature in the image corresponding to the first camera at the current moment; qx 2,i,j Represents the horizontal coordinate of the j-th pixel of the ith feature in the image corresponding to the second camera at the current moment.

4. The transparent display interaction method for augmented reality according to claim 2, characterized in that: The specific calculation formula for obtaining the pixel distance difference of each feature in the image corresponding to the second and third cameras at the current moment according to the ordinate of each pixel point of each feature and the number of pixels contained in each feature is: Among them, l 2,3,i Represents the pixel distance difference between the i-th feature in the image corresponding to the second and third cameras at the current moment, N i Indicates the number of pixels contained in the i-th feature; qy 2,i,j Represents the ordinate of the jth pixel of the ith feature in the image corresponding to the second camera at the current moment; qy 3,i,j Represents the ordinate of the j-th pixel of the ith feature in the image corresponding to the third camera at the current moment.

5. The transparent display interaction method for augmented reality according to claim 1, characterized in that: The specific steps of obtaining the virtual image at the current moment according to the gray value of each pixel in the real image at the current moment are as follows: According to the gray value of each pixel in the real image at the current moment, a final extended window of the touch point in the real image at the current moment is obtained; Record the boundary of the minimum circumscribed rectangle of the final extended window in the real image at the current moment as the optimal boundary of the touch point in the real image at the current moment; A virtual image at the current moment is generated according to an image within an optimal boundary of the touch point in the real image at the current moment.

6. A transparent display interaction method for augmented reality according to claim 5, characterized in that: The method of obtaining the final extended window of the touch point in the real image at the current moment according to the gray value of each pixel in the real image at the current moment includes the following specific steps: In the real image at the current moment, according to the preset window size, the initial window center is set as the coordinate of the touch point, and the window is increased cyclically in four directions in turn, up, down, left and right. The size of each expansion is the preset window size. When the absolute value of the difference between the grayscale mean of all pixels in the extended window and the grayscale mean of all pixels in the real image is greater than the absolute value of the difference between the grayscale mean of all pixels in the initial window and the grayscale mean of all pixels in the real image, the window expansion ends, and the final expanded window of the touch point in the real image at the current moment is obtained.

7. The transparent display interaction method for augmented reality according to claim 1, characterized in that: The step of obtaining the abscissa and ordinate of the dynamic display point of the virtual image at the next moment in the future according to the abscissa and ordinate of the dynamic display point of the virtual image at the current moment includes the following specific steps: Record the real images at the previous H moments adjacent to the real image at the current moment as reference images of the real image at the current moment, where H is a preset number threshold; The images corresponding to the three cameras at each moment and the corresponding real images are subjected to feature detection using a scale-invariant feature transformation algorithm to obtain the features of the real images at each moment; The pixel point corresponding to the mean of the horizontal and vertical coordinates of the central pixel points of all features in each reference image corresponding to the real image at the current moment is recorded as the key point of each reference image; Performing curve fitting on the key points in all reference images corresponding to the real image at the current moment, to obtain a fitting curve of the key points in all reference images corresponding to the real image at the current moment; According to the coordinates of the dynamic display points of the virtual image at the current moment, the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment, and the tangent slope of the key points on the fitting curve, the horizontal and vertical coordinates of the dynamic display points of the virtual image at the next moment in the future are obtained.

8. A transparent display interaction method for augmented reality according to claim 7, characterized in that: The specific calculation formula for obtaining the abscissa and ordinate of the dynamic display point of the virtual image at the next moment in the future according to the coordinates of the dynamic display point of the virtual image at the current moment, the abscissa of the key points in all reference images corresponding to the real image at the current moment, and the tangent slope of the key points on the fitting curve is: dx s+1 =x s +Δx s of s+1 =y s +k s ×Δx s Among them, dx s+1 The horizontal coordinate of the dynamic display point of the virtual image at the next moment in the future, dy s+1 Indicates the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, x s Indicates the horizontal coordinate of the dynamic display point of the virtual image at the current moment, y s The vertical coordinate of the dynamic display point of the virtual image at the current moment, Δx s k represents the average rate of change of the horizontal coordinates of the key points in all reference images corresponding to the real image at the current moment; s Represents the mean of the tangent slopes of the key points on the fitting curve in all reference images corresponding to the real image at the current moment.

9. The transparent display interaction method for augmented reality according to claim 1, characterized in that: The method of completing the augmented reality function of the virtual image at the next moment in the future according to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future includes the following specific steps: The horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the current moment are updated to the horizontal coordinate and the vertical coordinate of the dynamic display point of the virtual image at the next moment in the future, so as to obtain the display position of the virtual image at the next moment in the future; According to the display position of the virtual image at the next moment in the future, the virtual image at the next moment in the future is displayed on the display screen, completing the augmented reality function of the virtual image at the next moment in the future.

Images