Response mapping method and device of AR glasses cooperating with gesture operation terminal

Abstract

The invention discloses a response mapping method and device for operating a terminal through cooperation of AR glasses and gestures, space positions and gestures of a mobile phone terminal and the gestures are calculated in real time through a deep learning technology and a visual geometric algorithm, the space positions and gestures of the gestures are projected to a plane of the mobile phone terminal through a projection algorithm, and the position of a hand relative to the mobile phone terminal is obtained. And the position information and the screen picture of the mobile phone are uploaded to the AR equipment, so that the AR equipment can synchronously display the position of the finger of the user while displaying the picture of the mobile phone, and visual reference and help are provided for the operation of the user.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a response mapping method and device for an AR glasses-assisted gesture operation terminal. Background Art

[0002] Augmented Reality (AR) is a technology that combines the real world with virtual information based on real-time computer computing and multi-sensor fusion. This technology simulates and re-outputs human vision, hearing, smell, touch and other senses, and superimposes virtual information on real information, providing people with an experience beyond the real world.

[0003] As a popular entertainment interactive medium, AR provides more and more users with smooth and open audio-visual and gaming experiences. Currently, the most popular method is for users to wear AR glasses and display the screen of the mobile terminal in the AR glasses through a data cable. Users watch videos or game interfaces through the glasses, but the control operations are still retained on the mobile terminal side. Because in this scenario, users can only see the screen projected from the mobile phone through AR glasses and cannot see the real world scene, so users cannot see the real mobile terminal and the position of their hands. When controlling video playback or playing games, it will lead to "blind operation", which is easy to cause misoperation and affect the real experience.

[0004] In response to this problem, the researchers designed a processing device that collects finger images and the screen of a mobile phone terminal. However, they further discovered that when the spatial posture of the gesture is projected onto the plane of the mobile phone terminal through a projection algorithm to obtain the position of the hand relative to the mobile phone terminal, the real-time coordinate position of the thumb cannot be tracked. At the same time, the posture of the thumb obtained by projection is relatively rigid, and the position of the thumb above the mobile phone screen will cause deviations in the coordinates of the thumb projection during the projection calculation process due to the size of the mobile phone screen. Summary of the invention

[0005] To address this pain point, the present invention proposes a vision-based terminal-gesture positioning and tracking algorithm and device, which uses deep learning technology and visual geometry algorithms to calculate the spatial position and posture of the mobile phone terminal and gesture in real time, and then uses a projection algorithm to project the spatial position of the gesture onto the plane of the mobile phone terminal to obtain the position of the hand relative to the mobile phone terminal. This position information is uploaded to the AR device together with the mobile phone screen image, so that the AR device can display the position of the user's finger while displaying the mobile phone screen, providing a visual reference and help for the user's operation.

[0006] The purpose of the present invention is to provide a response mapping method and device for AR glasses in collaboration with a gesture operation terminal, which solves the above-mentioned technical problems pointed out in the prior art.

[0007] The present invention provides a response mapping method for an AR glasses collaborative gesture operation terminal, comprising the following operation steps: acquiring a binocular view mobile phone image containing a complete mobile phone outline, performing corner point detection on the acquired binocular view mobile phone image to obtain the coordinates of the mobile phone corner points, and then using the mobile phone corner point coordinates to obtain the mobile phone corner point space coordinates by triangulation and plane fitting; the binocular view mobile phone image includes a left-eye view mobile phone image and a right-eye view mobile phone image; acquiring a binocular view thumb image containing a thumb screen, performing corner point detection on the acquired thumb image to obtain the coordinates of the thumb corner points, and then using the thumb corner point coordinates to obtain the space coordinates of the mobile phone corner points; The thumb corner spatial coordinates and thumb posture are obtained by processing respectively through triangulation, Kalman filtering algorithm and ICP algorithm; the thumb image includes a thumb image of a left eye view and a thumb image of a right eye view; the directional dimension ratio of the thumb projected onto the mobile phone screen is calculated based on the thumb corner spatial coordinates and the mobile phone corner spatial coordinates; the directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y; the binocular view mobile phone image is transmitted to an AR device to obtain an AR display image; the thumb image is fused with the AR display image based on the directional dimension ratio and the thumb posture to obtain a fused image.

[0008] Preferably, the mobile phone corner point coordinates are obtained after corner point detection of the collected binocular view mobile phone image, and then the mobile phone corner point coordinates are used to obtain the mobile phone corner point spatial coordinates through triangulation and plane fitting, including the following operating steps: corner point detection is performed on the binocular view mobile phone image through a pre-trained key point detection network model to obtain the mobile phone corner point coordinates corresponding to the binocular view mobile phone image; the mobile phone corner point coordinates are calculated by a triangulation algorithm using internal and external parameters of the camera to obtain the three-dimensional coordinates corresponding to the mobile phone corner point coordinates; the three-dimensional coordinates are plane fitted to obtain the mobile phone corner point spatial coordinates.

[0009] Preferably, the pre-trained key point detection network model is obtained by training based on a YOLO model; and obtaining the pre-trained key point detection network model includes the following steps:

[0010] A plurality of sample mobile phone outline images are obtained by crawling through a web crawler method; a preprocessing operation is performed on the sample mobile phone outline images to obtain preprocessed sample mobile phone outline images; the preprocessing operation includes a screen size adjustment operation; the preprocessed sample mobile phone outline images are labeled to obtain a sample data set; an initial key point detection network model is established; the initial key point detection network model is trained based on the sample data set, and the trained key point detection network model is output after the initial key point detection network model converges.

[0011] Preferably, the plane fitting of the three-dimensional coordinates to obtain the spatial coordinates of the mobile phone corners is to first determine the general expression of the prior plane through the average point coordinates of each three-dimensional coordinate, then construct a parameter-free prior plane through the average point coordinates, then solve the plane equation parameters of the general expression of the prior plane, and finally obtain the general expression of the target prior plane, and then project the three-dimensional coordinates onto the target prior plane to obtain the spatial coordinates of the mobile phone corners; specifically including the following steps:

[0012] Calculate the average point coordinates P_m (x_m, y_m, z_m) based on each of the three-dimensional coordinates;

[0013] Constructing a priori plane based on the average point coordinates P_m(x_m, y_m, z_m);

[0014] An error matrix H is constructed based on the error terms from each of the three-dimensional coordinates to the priori plane;

[0015] The error matrix H is:

[0016]

[0017] Where (x1, y1, z1), (x2, y2, z2), (x3, y3, z3) and (x4, y4, z4) represent the three-dimensional coordinates corresponding to the corner coordinates of the mobile phone; (x m ,y m , z m ) represents the average coordinate of the four three-dimensional coordinates;

[0018] Performing a singular value decomposition operation on the error matrix H to obtain a decomposition matrix H';

[0019] The decomposition matrix H' is:

[0020] H'=UDV T ;

[0021] Where U is the left singular vector matrix, D is the singular value diagonal matrix, V T is the right singular vector matrix;

[0022] Get the right singular vector matrix V T The last column of eigenvectors is used as plane parameter A, plane parameter B, and plane parameter C;

[0023] Calculate the plane parameter D according to the three-dimensional coordinates and the priori plane;

[0024] Acquire a target priori plane according to the plane parameter A, the plane parameter B, the plane parameter C and the plane parameter D;

[0025] The general expression of the target prior plane is:

[0026] Ax_m+By_m+Cz_m+D=0;

[0027] In the formula, A, B, C, and D are the plane equation parameters respectively;

[0028] Each of the three-dimensional coordinates is vertically projected onto the target a priori plane to obtain the spatial coordinates of the mobile phone corner points corresponding to each of the three-dimensional coordinates.

[0029] Preferably, the method of using the thumb corner point coordinates to process the thumb corner point spatial coordinates and thumb posture through triangulation, Kalman filter algorithm and ICP algorithm in sequence comprises the following steps: obtaining camera internal and external parameters; calculating initial thumb corner point three-dimensional coordinates through triangulation method according to the camera internal and external parameters and the thumb corner point coordinates; obtaining frame thumb corner point coordinates of continuous frames; calculating the frame thumb corner point coordinates through triangulation method to obtain frame thumb corner point three-dimensional coordinates corresponding to each frame; predicting and correcting the frame thumb corner point three-dimensional coordinates through Kalman filter algorithm to obtain corrected frame thumb corner point three-dimensional coordinates; calculating the thumb posture corresponding to each corrected frame thumb corner point three-dimensional coordinates through ICP algorithm based on the initial thumb corner point three-dimensional coordinates and the corrected frame thumb corner point three-dimensional coordinates;

[0030] The thumb gesture includes a rotation matrix R and a translation vector T.

[0031] Preferably, the calculating of the initial three-dimensional coordinates of the thumb corner points by triangulation method according to the camera internal and external parameters and the coordinates of the thumb corner points comprises the following steps:

[0032] Obtain the first thumb corner point coordinates of the thumb corner point in N frames when the mobile phone screen surface is in a stationary state;

[0033] The first thumb corner point coordinates are calculated by triangulation according to the internal and external parameters of the camera to obtain the first thumb corner point three-dimensional coordinates; the average coordinates of the first thumb corner point three-dimensional coordinates corresponding to each of the thumb corner point coordinates are calculated; and the average coordinates are determined as the initial thumb corner point three-dimensional coordinates.

[0034] Preferably, the predicting and correcting operation of the frame thumb corner point three-dimensional coordinates by using a Kalman filter algorithm to obtain the corrected frame thumb corner point three-dimensional coordinates includes the following operation steps:

[0035] Establishing a motion model for the three-dimensional coordinates of the thumb corner points in each frame;

[0036] Traversing the three-dimensional coordinates of the thumb corner points of each frame, predicting the three-dimensional coordinates of the thumb corner points of the frame corresponding to the three-dimensional coordinates of the thumb corner points of the continuous frames and the motion model, and obtaining the predicted three-dimensional coordinates corresponding to the three-dimensional coordinates of the thumb corner points of the frame of the t-th frame;

[0037] Correcting the predicted three-dimensional coordinates based on the three-dimensional coordinates of the thumb corner point in frame t-1 to obtain corrected three-dimensional coordinates of the thumb corner point;

[0038] Repeat the above operation until all the three-dimensional coordinates of the thumb corner points of the frames are traversed.

[0039] Preferably, the step of calculating the thumb postures corresponding to the corrected frame thumb corner point three-dimensional coordinates based on the initial thumb corner point three-dimensional coordinates and the corrected frame thumb corner point three-dimensional coordinates by an ICP algorithm comprises the following steps:

[0040] Calculate and obtain the initial centroid coordinates of each of the initial thumb corner point three-dimensional coordinates and the frame centroid coordinates of each of the corrected frame thumb corner point three-dimensional coordinates;

[0041] The initial relative centroid coordinates are calculated based on the initial centroid coordinates and the three-dimensional coordinates of each initial thumb corner point; and the frame relative centroid coordinates are calculated based on the corrected three-dimensional coordinates of the thumb corner point and the frame centroid coordinates;

[0042] The initial relative centroid coordinates are calculated as follows:

[0043] Initial relative centroid coordinates = initial thumb corner three-dimensional coordinates - initial centroid coordinates;

[0044] The frame relative centroid coordinates are calculated as follows:

[0045] Frame relative centroid coordinates = corrected thumb corner three-dimensional coordinates - frame centroid coordinates;

[0046] Normalizing each of the initial relative centroid coordinates to obtain an initial normalized point set; normalizing each of the frame relative centroid coordinates to obtain a frame normalized point set;

[0047] Based on the initial normalized point set and the frame normalized point set, the thumb posture (R, T) corresponding to each corrected frame thumb corner point three-dimensional coordinate is calculated by the least square method;

[0048] The thumb posture (R, T) is calculated as follows:

[0049]

[0050] Where R is the rotation matrix; T is the translation vector; argmin is the input value corresponding to the minimum value of the function; sp_i is the i-th initial relative centroid coordinate in the initial normalized point set; dp_i is the i-th frame relative centroid coordinate in the frame normalized point set.

[0051] Preferably, the step of calculating the directional dimension ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points comprises the following steps:

[0052] The spatial coordinates of each thumb corner point are respectively vertically projected onto the spatial intersection points in the target a priori plane;

[0053] The spatial intersection points represent the intersection points of the spatial coordinates of each thumb corner point projected onto the target a priori plane and the target a priori plane;

[0054] Calculate and obtain the left Euclidean distance dist_x of each of the spatial intersection points perpendicular to the left edge of the target a priori plane and the upper Euclidean distance dist_y of each of the spatial intersection points perpendicular to the upper edge of the target a priori plane;

[0055] Calculating the length L of the target a priori plane and the width W of the target a priori plane based on the spatial coordinates of each of the mobile phone intersection points;

[0056] The ratio r_X in the X direction is calculated based on the left Euclidean distance dist_x and the length L of the target a priori plane; the ratio r_Y in the Y direction is calculated based on the upper Euclidean distance dist_y and the width W of the target a priori plane;

[0057] The calculation method of the ratio r_X in the X direction is:

[0058] r_X=dist_x÷L;

[0059] The ratio r_Y in the Y direction is calculated as follows:

[0060] r_Y=dist_y÷W.

[0061] Accordingly, the present invention also proposes a response mapping device for an AR glasses-coordinated gesture operation terminal, comprising a collection and calculation unit, a calculation unit, and a fusion module;

[0062] Wherein, the acquisition and calculation unit includes a binocular vision module and a calculation module;

[0063] Wherein, the binocular vision module is used to obtain a binocular view mobile phone image containing a complete mobile phone outline;

[0064] The calculation module is used to obtain the coordinates of the corner points of the mobile phone after corner point detection of the collected binocular view mobile phone image, and then use the coordinates of the mobile phone corner points to obtain the spatial coordinates of the mobile phone corner points by triangulation and plane fitting;

[0065] The binocular view mobile phone image includes a left-view mobile phone image and a right-view mobile phone image;

[0066] The binocular vision module is also used to obtain a thumb image containing a thumb picture;

[0067] The calculation module is further used to obtain the coordinates of the thumb corner points after corner point detection of the collected thumb image, and then use the coordinates of the thumb corner points to process through triangulation, Kalman filter algorithm and ICP algorithm in sequence to obtain the spatial coordinates of the thumb corner points and the thumb posture;

[0068] The thumb image includes a left-eye view thumb image and a right-eye view thumb image;

[0069] The calculation unit is used to calculate the directional size ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points;

[0070] The directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y;

[0071] The fusion module is used to transmit the binocular view mobile phone image to the AR device to obtain an AR display image; based on the direction size ratio and the thumb posture, the thumb image is fused with the AR display image to obtain a fused image.

[0072] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0073] By analyzing the response mapping method and device of the AR glasses cooperative gesture operation terminal provided by the present invention, it can be known that in specific applications, the accurate position of the mobile phone screen in the three-dimensional space is first determined by the spatial coordinates of the mobile phone corner points, so as to provide a position basis for the mobile phone screen for subsequent steps; by acquiring an image containing the thumb, the two-dimensional corner point coordinates of the thumb are obtained through corner point detection, and then these two-dimensional corner point coordinates are processed by triangulation, Kalman filtering and ICP algorithm to obtain the three-dimensional spatial coordinates and posture of the thumb, accurately track the position and posture of the thumb, and provide data support for fusing and displaying the actual position of the thumb; based on the spatial coordinates of the mobile phone corner points and the spatial coordinates of the thumb corner points, the projection size ratio of the thumb on the mobile phone screen (the size ratio in the X direction and the Y direction) is calculated, and the spatial coordinates of the thumb are converted into the relative size on the mobile phone screen, so as to ensure that the relative position of the thumb is accurately displayed on the AR device; the binocular view mobile phone image is transmitted to the AR device, and based on the directional size ratio and the thumb posture, the thumb image is fused with the AR display image, and finally a fused image is generated, and then the synthesized image is displayed on the AR device, so as to realize the synchronous mapping of the binocular view mobile phone image and the thumb image, and provide an augmented reality experience of user interaction;

[0074] Through deep learning technology and visual geometry algorithms, the spatial position and posture of the mobile terminal and gestures are calculated in real time, and then the spatial position of the gestures is projected onto the plane of the mobile terminal through a projection algorithm to obtain the position of the hand relative to the mobile terminal. This position information is uploaded to the AR device together with the mobile phone screen image, so that the AR device can display the position of the user's fingers while displaying the mobile phone screen, providing visual reference and help for the user's operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A schematic diagram of the overall operation steps of a response mapping method for an AR glasses cooperative gesture operation terminal provided in the first embodiment of the present invention;

[0077] Figure 2 A thumb corner simulation schematic diagram of a response mapping method for an AR glasses-assisted gesture operation terminal provided in the first embodiment of the present invention;

[0078] Figure 3A schematic diagram of the operation steps of obtaining the spatial coordinates of the thumb corner point and the thumb posture of a response mapping method of an AR glasses cooperative gesture operation terminal provided in the first embodiment of the present invention;

[0079] Figure 4 A schematic diagram of the simulation of the Euclidean distance from the spatial intersection point perpendicular to the left edge of the mobile phone plane and the Euclidean distance from the upper edge of the mobile phone plane in a response mapping method of an AR glasses cooperative gesture operation terminal provided in the first embodiment of the present invention;

[0080] Figure 5 A schematic diagram of the overall architecture of a response mapping device for an AR glasses-assisted gesture operation terminal provided in Embodiment 2 of the present invention.

[0081] Figure numerals: acquisition and calculation unit 10; calculation unit 20; fusion module 30; binocular vision module 11; calculation module 12. DETAILED DESCRIPTION

[0082] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] The present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0084] Embodiment 1

[0085] like Figure 1 As shown, the present invention proposes a response mapping method for AR glasses in collaboration with a gesture operation terminal, comprising the following steps:

[0086] Step S10: (shooting by binocular vision module) obtaining a binocular view mobile phone image containing a complete outline of the mobile phone, obtaining the mobile phone corner point coordinates after corner point detection of the collected binocular view mobile phone image, and then using the mobile phone corner point coordinates to obtain the mobile phone corner point spatial coordinates by triangulation and plane fitting;

[0087] The binocular view mobile phone image includes a left-view mobile phone image and a right-view mobile phone image;

[0088] It should be noted that the present invention only considers mobile phone terminals with rectangular outlines, and does not consider special-shaped mobile phone terminals.

[0089] Step S20: (shooting by binocular vision module) obtaining a thumb image containing a thumb picture, performing corner point detection on the acquired thumb image to obtain the coordinates of the thumb corner points, and then using the thumb corner point coordinates to process by triangulation, Kalman filter algorithm and ICP algorithm in sequence to obtain the spatial coordinates of the thumb corner points and the thumb posture;

[0090] The thumb image includes a left-eye view thumb image and a right-eye view thumb image (i.e., a left thumb image photographed by a left-eye vision module and a right thumb image photographed by a right-eye vision module in a binocular vision module, and the left thumb image and the right thumb image are essentially two images of the same thumb photographed by the left and right vision modules of the binocular vision module);

[0091] It should be noted that, when a user plays games on a mobile phone terminal, the user generally uses the tip of the thumb to control the screen, and the remaining fingers are used to fix the phone. Therefore, the present invention is limited to tracking the spatial coordinates of the thumb corners of the left and right hands, and the phalangeal joints of the thumbs; Figure 2 As shown, we define four corner points with obvious features of the nail as the key points of the thumb phalanx joint;

[0092] The thumb corner point coordinates in the above embodiment of the present application are obtained by performing a corner point detection operation using a pre-trained thumb corner point detection model. The pre-trained thumb corner point detection model in the above embodiment of the present application is obtained by training based on the YOLO model.

[0093] The first thing is to get the image coordinates of these four corner points in the binocular image. To achieve this goal, we collected thumb images of multiple people through the binocular vision module, and manually annotated the image coordinates of the four key points of each thumb to obtain the thumb key point dataset.

[0094] Next, we use the pre-trained thumb corner detection model (such as YOLO) to train the thumb key point dataset generated in the previous step to obtain the thumb corner detection model. Based on this model, we can automatically detect the thumb corner coordinates of any thumb image in the scene (the thumb corner coordinates are the image 2D coordinates of the 4 key points of the thumb, Left_Thumb_Coords and Right_Thumb_Coords, that is, the image 2D coordinates of the 4 key points of the left thumb image and the image 2D coordinates of the 4 key points of the right thumb image).

[0095] Step S30: Calculating the directional dimension ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points;

[0096] The directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y;

[0097] It should be noted that through the above steps, the position and plane equation of the mobile phone plane in the spatial coordinate system, as well as the position and posture of the thumb have been obtained. However, in the AR device, the visualization is performed according to the pixel size of the mobile phone screen. Therefore, it is necessary to calculate the size ratio of the thumb in the X and Y directions after it is projected onto the mobile phone screen.

[0098] Step S40: transferring the binocular view mobile phone image to the AR device to obtain an AR display image (i.e., an AR device display image); fusing the thumb image with the AR display image based on the direction size ratio and the thumb posture to obtain a fused image;

[0099] Then, the fused image, i.e., the fused image, is used as the final display screen, and a display screen in which the binocular view mobile phone image and the thumb image are synchronously mapped is displayed on the AR device.

[0100] It should be noted that, in the above-mentioned embodiment of the present application, in the AR device, the screen of the mobile phone terminal is first transmitted losslessly to the screen of the AR device through a signal transmission technology such as projection, so as to obtain an AR display image on the screen of the AR device; further, the ratios r_X and r_Y of the thumb key points (thumb images) calculated above within the range of the mobile phone screen are multiplied by the pixel resolution of the mobile phone screen to obtain the coordinates of the landing point on the mobile phone screen, which are visualized by rendering or other means (visualization means obtaining the above-mentioned fused image on the screen of the AR device); or, a visual digital model of the thumb is prefabricated through tools such as Unity, and the thumb postures R and T of the t-th frame calculated above are used to drive the model to be visualized in the interface (visualization means obtaining the above-mentioned fused image on the screen of the AR device);

[0101] The above-mentioned embodiment of the present application first determines the accurate position of the mobile phone screen in three-dimensional space through the spatial coordinates of the mobile phone corner points, providing a position basis for the mobile phone screen for subsequent steps; by acquiring an image containing the thumb, the two-dimensional corner coordinates of the thumb are obtained through corner detection, and then these two-dimensional corner coordinates are processed through triangulation, Kalman filtering and ICP algorithm to obtain the three-dimensional spatial coordinates and posture of the thumb, accurately track the position and posture of the thumb, and provide data support for the fusion and display of the actual position of the thumb; based on the spatial coordinates of the mobile phone corner points and the spatial coordinates of the thumb corner points, the projection size ratio of the thumb on the mobile phone screen (the size ratio in the X direction and the Y direction) is calculated, and the spatial coordinates of the thumb are converted into the relative size on the mobile phone screen to ensure that the relative position of the thumb is accurately displayed on the AR device; the binocular view mobile phone image (binocular view mobile phone image or mobile phone screen image) is transmitted to the AR device, and based on the direction size ratio and the thumb posture, the thumb image is fused with the AR display image to finally generate a fused image, and then the synthesized image is displayed on the AR device to achieve synchronous mapping of the mobile phone screen and the thumb image, providing an augmented reality experience of user interaction;

[0102] Through deep learning technology and visual geometry algorithms, the spatial position and posture of the mobile terminal and gestures are calculated in real time, and then the spatial position of the gestures is projected onto the plane of the mobile terminal through a projection algorithm to obtain the position of the hand relative to the mobile terminal. This position information is uploaded to the AR device together with the mobile phone screen image, so that the AR device can display the position of the user's fingers while displaying the mobile phone screen, providing visual reference and help for the user's operation.

[0103] Specifically, in step S10, the mobile phone corner point coordinates are obtained after corner point detection of the collected binocular view mobile phone image, and then the mobile phone corner point coordinates are used to obtain the mobile phone corner point space coordinates by triangulation and plane fitting, including the following operation steps:

[0104] Step S11: performing corner point detection on the binocular view mobile phone image through a pre-trained key point detection network model to obtain the mobile phone corner point coordinates corresponding to the binocular view mobile phone image;

[0105] The mobile phone corner point coordinates include the mobile phone corner point coordinates of the mobile phone image in the left eye view and the mobile phone corner point coordinates of the mobile phone image in the right eye view;

[0106] It should be noted that the above-mentioned mobile phone corner point coordinates (i.e., the upper left corner point coordinates, lower left corner point coordinates, upper right corner point coordinates and lower right corner point coordinates corresponding to the left binocular view mobile phone image and the upper left corner point coordinates, lower left corner point coordinates, upper right corner point coordinates and lower right corner point coordinates corresponding to the right binocular view mobile phone image) are the 2D spatial positions of the four corner points of the binocular view mobile phone image (i.e., the 2D coordinates on the binocular view mobile phone image);

[0107] Step S12: using the internal and external parameters of the camera to calculate the coordinates of the corner points of the mobile phone through a triangulation algorithm to obtain the three-dimensional coordinates corresponding to the coordinates of the corner points of the mobile phone;

[0108] The three-dimensional coordinates include Phone_Pt1 (x1, y1, z1), Phone_Pt2 (x2, y2, z2), Phone_Pt3 (x3, y3, z3), Phone_Pt4 (x4, y4, z4);

[0109] It should be noted that the above-mentioned embodiment of the present application obtains the coordinates of the corner points of the mobile phone in the left and right images based on the internal and external parameters of the binocular camera and the model detection in the previous step, and calculates the 3D positions of the four corner points of the mobile phone in the world coordinate system through the triangulation algorithm; the triangulation algorithm is a commonly used method in spatial geometric operations, which is based on the observation of the same point in space by cameras at multiple different positions. The spatial position of the point can be calculated through the pixel coordinates of the point in the imaging plane of each camera, and the motion relationship between the cameras; the above-mentioned triangulation algorithm is a prior art and will not be repeated in this application.

[0110] Step S13: performing plane fitting on the three-dimensional coordinates to obtain the spatial coordinates of the corner points of the mobile phone;

[0111] It should be noted that due to the influence of noise, the spatial positions of the four corner points obtained by triangulation will have errors. It is known that the four corner points of the mobile phone are coplanar in space (space refers to the same mobile phone screen plane, that is, coplanar on the 3D plane). Therefore, we perform plane fitting on the results calculated in the previous step to obtain the spatial coordinates of the mobile phone corner points.

[0112] Specifically, in step S11, the pre-trained key point detection network model is trained based on the YOLO model; the acquisition of the pre-trained key point detection network model includes the following operation steps:

[0113] Step S111: crawling a plurality of sample mobile phone outline images by a web crawler method; performing a preprocessing operation on the sample mobile phone outline images to obtain preprocessed sample mobile phone outline images;

[0114] The pre-processing operation includes a picture size adjustment operation (i.e., a size adjustment operation of the sample mobile phone outline image);

[0115] It should be noted that the above embodiment of the present application obtains a large number of front-facing pictures of different mobile phone models by means of a web crawler, and first converts each picture into the image resolution size we need by means of resizing (adjusting the size) or adding padding (adding a border), so as to provide an initial data basis for the subsequent training of the key point detection network model;

[0116] Step S112: annotating the preprocessed sample mobile phone outline image to obtain a sample data set;

[0117] It should be noted that the above-mentioned annotation refers to manually annotating the sample mobile phone outline image with the coordinates of the mobile phone corner points of each sample mobile phone outline image; the above-mentioned embodiment of the present application annotates the image coordinates of the four corner points of the mobile phone for each picture through the manual annotation method, thereby generating a mobile phone data set, which is a direct data basis for the training of the subsequent key point detection network model;

[0118] Step S113: establishing an initial key point detection network model; training the initial key point detection network model based on the sample data set, and outputting the trained key point detection network model after the initial key point detection network model converges;

[0119] It should be noted that the above embodiment of the present application uses a deep learning key point detection network (such as the YOLO model) and uses the mobile phone data set generated in the previous step for model training to obtain a deep learning network model that can be used to detect the four corner points of the mobile phone. Using this model, the image of the mobile phone taken by the binocular module can be automatically detected to obtain the 2D coordinates of the four corner points of the mobile phone in the left and right images;

[0120] The above-mentioned operations of training the initial key point detection network model based on the sample data set and the model convergence judgment conditions are common knowledge to those skilled in the art and will not be elaborated in this application.

[0121] Specifically, in step S13, plane fitting is performed on the three-dimensional coordinates to obtain the spatial coordinates of the corner points of the mobile phone, including the following operation steps:

[0122] Step S131: Calculate the average point coordinates P_m (x_m, y_m, z_m) based on each of the three-dimensional coordinates;

[0123] It should be noted that the three-dimensional coordinates of the four corner points obtained in the above-mentioned embodiment of the present application are respectively: Phone_Pt1 (x1, y1, z1), Phone_Pt2 (x2, y2, z2), Phone_Pt3 (x3, y3, z3), Phone_Pt4 (x4, y4, z4); and the four corner points actually belong to the same plane, but the equation of the plane is unknown. Therefore, the embodiment of the present application calculates the average point coordinates of the three-dimensional coordinates of the four corner points, and then uses the three-dimensional coordinates of the four corner points and the average point coordinates to determine an a priori plane (the a priori plane is real, but its plane equation is unknown), and then uses the a priori plane for analysis to obtain the spatial coordinates of the corner points, as shown in the following steps.

[0124] Step S132: constructing a priori plane based on the average point coordinates P_m (x_m, y_m, z_m) (first determine the general expression of the priori plane through the average point coordinates, then construct a priori plane with unknown parameters through the average point coordinates, then solve the plane equation parameters of the general expression of the priori plane, and finally obtain the general expression of the target priori plane, and then project the three-dimensional coordinates onto the target priori plane to obtain the spatial coordinates of the mobile phone corner points);

[0125] It should be noted that the above-mentioned embodiment of the present application is a priori plane (expression of the priori plane) constructed by the general expression of the plane in space and the average point coordinates P_m (x_m, y_m, z_m). Assuming that the general expression of the plane in space is: A1x+B1y+C1z+D1=0, the positions of the four corner points obtained in the above steps are: Phone_Pt1(x1, y1, z1), Phone_Pt2(x2, y2, z2), Phone_Pt3(x3, y3, z3), Phone_Pt4(x4, y4, z4); at the same time, because the priori plane passes through the average point P_m(x_m, y_m, z_m) of all points, it satisfies the following equation (that is, the general expression of the priori plane is): A2x_m+B2y_m+C2z_m+D2=0;

[0126] However, in the above-mentioned embodiment of the present application, the plane equation parameters A1, B1, C1, D1 of the general expression of the plane in the assumed space and the plane equation parameters A2, B2, C2, D2 of the general expression of the constructed prior plane are unknown. Therefore, the embodiment of the present application also needs to obtain the plane equation parameters of the general expression of the prior plane to obtain the general expression of the target prior plane. The specific operations are shown in the following steps.

[0127] Step S133: constructing an error matrix H based on the error terms from each of the three-dimensional coordinates to the a priori plane (or a general expression of the a priori plane);

[0128] The error matrix H is:

[0129]

[0130] Where (x1, y1, z1), (x2, y2, z2), (x3, y3, z3) and (x4, y4, z4) represent the three-dimensional coordinates corresponding to the corner coordinates of the mobile phone; (x m ,y m , z m ) represents the average coordinate of the four three-dimensional coordinates;

[0131] Step S134: performing a singular value decomposition operation on the error matrix H to obtain a decomposition matrix H';

[0132] The decomposition matrix H' is:

[0133] H'=UDV T ;

[0134] Where U is the left singular vector matrix, D is the singular value diagonal matrix, V T is the right singular vector matrix;

[0135] It should be noted that the error terms from each three-dimensional coordinate to the prior plane obtained in the above embodiment of the present application can be constructed into an equation, namely:

[0136]

[0137] Then, the embodiment of the present application converts the equation As the matrix H, the plane parameters of the prior plane are obtained, thereby providing a data basis for the subsequent acquisition of the spatial coordinates of the corner points of each three-dimensional coordinate;

[0138] Step S135: Obtain the right singular vector matrix V T The last column of eigenvectors is used as plane parameter A, plane parameter B, and plane parameter C;

[0139] Step S136: Calculate the plane parameter D according to the three-dimensional coordinates and the priori plane;

[0140] The above-mentioned embodiment of the present application substitutes the three-dimensional coordinates into the general expression of the a priori plane obtained above (known plane parameter A, plane parameter B, plane parameter C), thereby calculating the plane parameter D.

[0141] Step S137: obtaining a target priori plane according to the plane parameter A, the plane parameter B, the plane parameter C and the plane parameter D;

[0142] The general expression of the target prior plane is:

[0143] Ax_m+By_m+Cz_m+D=0;

[0144] In the formula, A, B, C, and D are the plane equation parameters respectively;

[0145] Step S138: vertically projecting each of the three-dimensional coordinates onto the target a priori plane to obtain the spatial coordinates of the mobile phone corner points corresponding to each of the three-dimensional coordinates;

[0146] The spatial coordinates of the mobile phone corner points include Phone_Pt1_proj, Phone_Pt2_proj, Phone_Pt3_proj, and Phone_Pt4_proj.

[0147] It should be noted that, in the above embodiment of the present application, the coordinates of the point where the three-dimensional coordinates are vertically projected onto the above target a priori plane (or the coordinates of the intersection of the three-dimensional coordinates and the a priori plane) are taken as the spatial coordinates of the corner point;

[0148] The above embodiment of the present application constructs a general expression of an initial plane (i.e., the general expression of the a priori plane) by the average point coordinates of each three-dimensional coordinate, and then constructs an error matrix H according to the error terms from each three-dimensional coordinate to the initial plane, performs a singular value decomposition operation on the error matrix, and obtains a decomposition matrix H'; then, according to the right singular vector matrix V in the decomposition matrix T The plane parameters A, B, and C in the general expression of the initial plane are obtained, and then the known three-dimensional coordinates are substituted into the general expression of the initial plane of the known plane parameters A, B, and C to calculate the plane parameter D, thereby obtaining the general expression of the final target a priori plane; and then the vertical projection points (i.e., projection points) of each three-dimensional coordinate and the target a priori plane are finally used as the spatial coordinates of the mobile phone corner points;

[0149] Due to the influence of noise, the three-dimensional coordinates corresponding to the four mobile phone corner point coordinates obtained by triangulation will have errors in the spatial position. Since the four mobile phone corner point coordinates of the known mobile phone are analyzed and identified from the binocular view mobile phone image, their corresponding three-dimensional coordinates are coplanar in space. Therefore, the embodiment of the present application performs plane fitting on the result calculated in the previous step to obtain the plane corresponding to the four three-dimensional coordinates (that is, the above-mentioned target prior plane); further, by projecting the three-dimensional coordinates onto the target prior plane (this operation is essentially a correction of the four three-dimensional coordinates), the spatial coordinates of the mobile phone corner points corresponding to the four mobile phone corner point coordinates that are coplanar in space are obtained.

[0150] Specifically, Figure 3 As shown, in step S20, the thumb corner point coordinates are processed by triangulation, Kalman filter algorithm and ICP algorithm in sequence to obtain the thumb corner point spatial coordinates and thumb posture, including the following operation steps:

[0151] Step S21: obtaining camera internal and external parameters; calculating the initial three-dimensional coordinates of the thumb corner point through triangulation method according to the camera internal and external parameters and the coordinates of the thumb corner point;

[0152] It should be noted that the above-mentioned camera internal and external parameters are the camera internal and external parameters of the binocular vision module;

[0153] The above-mentioned embodiment of the present application obtains the 2D coordinates of the four thumb key points in the left and right images according to the internal and external parameters of the binocular camera and the model detection in the previous step, and calculates the 3D position in the world coordinate system through the triangulation algorithm.

[0154] Step S22: acquiring frame thumb corner coordinates of continuous frames; calculating the frame thumb corner coordinates by triangulation to obtain the frame thumb corner three-dimensional coordinates corresponding to each frame; predicting and correcting the frame thumb corner three-dimensional coordinates by Kalman filtering algorithm to obtain corrected frame thumb corner three-dimensional coordinates (i.e. the above-mentioned thumb corner spatial coordinates);

[0155] Note that the initial thumb corner 3D coordinates, frame thumb corner coordinates and frame thumb corner 3D coordinates are different concepts; the initial thumb corner 3D coordinates and frame thumb corner 3D coordinates are important parameters for constituting thumb gestures;

[0156] It should be noted that the above embodiment of the present application records the coordinates of the thumb corner points of each frame under the continuous frames of the thumb (or the coordinates of the thumb corner points) performing normal operation movements, and then continues to calculate using the triangulation method to obtain the three-dimensional space coordinates corresponding to the coordinates of the thumb corner points of each frame; assuming that the three-dimensional space coordinates of the four key points (the four key points, namely the four thumb corner point coordinates) of the left thumb are calculated for the t-th frame as follows:

[0157] Left_Thumb_Pt0,Left_Thumb_Pt1,Left_Thumb_Pt2,Left_Thumb_Pt3;

[0158] Then, the three-dimensional spatial coordinates of the four thumb corner point coordinates are further predicted and corrected by the Kalman filter algorithm to obtain the corrected three-dimensional coordinates of the frame thumb corner points. The specific operations are shown in the following steps.

[0159] Step S23: Based on the initial three-dimensional coordinates of the thumb corner points and the corrected three-dimensional coordinates of the frame thumb corner points, the thumb postures corresponding to the corrected three-dimensional coordinates of the frame thumb corner points are calculated by the ICP algorithm;

[0160] The thumb gesture includes a rotation matrix R and a translation vector T.

[0161] It should be noted that the above-mentioned embodiment of the present application obtains the camera internal and external parameters of the binocular vision module and the 2D coordinates of the thumb corner point, and uses the triangulation method to calculate the three-dimensional coordinates of the initial thumb corner point, thereby providing a basis for the initial position of the thumb in the world coordinate system for subsequent posture calculation; further, by obtaining the coordinates of the thumb corner points in consecutive frames, and calculating the three-dimensional coordinates of each frame by triangulation, the Kalman filter algorithm is used to predict and correct these three-dimensional coordinates to obtain more accurate and stable three-dimensional coordinates of the thumb, so that the determination of the thumb position is more accurate; and then, by using the ICP algorithm to calculate the posture of the thumb based on the initial three-dimensional coordinates and the corrected frame three-dimensional coordinates, accurate posture information of the thumb is provided, which provides a basis for correctly displaying the position and direction of the thumb in the AR device.

[0162] Specifically, in step S21, the initial three-dimensional coordinates of the thumb corners are calculated by triangulation according to the camera internal and external parameters and the coordinates of the thumb corners, including the following steps:

[0163] Step S211: obtaining the first thumb corner point coordinates of the thumb corner point in N frames when the mobile phone screen surface is in a stationary state;

[0164] Step S212: Calculate the first thumb corner point coordinates by triangulation according to the internal and external parameters of the camera to obtain the first thumb corner point three-dimensional coordinates; calculate the average coordinates of the first thumb corner point three-dimensional coordinates corresponding to each of the thumb corner point coordinates; determine the average coordinates as the initial thumb corner point three-dimensional coordinates;

[0165] It should be noted that, in the above-mentioned embodiment of the present application, the user's thumb is first placed flat on the surface of the mobile phone screen and remains still for N frames (i.e., the coordinates of each thumb corner point remain still on the surface of the mobile phone screen for N frames) through interface guidance, and the coordinates of the first thumb corner point of the four key points of the thumb in each frame are recorded in the N frames, and then the three-dimensional coordinates corresponding to the first thumb corner point coordinates (i.e., the above-mentioned first thumb corner point three-dimensional coordinates) are calculated by using the triangulation method, and the average value of the results (i.e., the above-mentioned average coordinates, the average coordinates are three-dimensional coordinates) is calculated as the initial position of the thumb (i.e., the above-mentioned initial thumb corner point three-dimensional coordinates);

[0166] For example, to get the initial position of the left thumb: Left_Thumb_Init_Pt0, Left_Thumb_Init_Pt1, Left_Thumb_Init_Pt2, Left_Thumb_Init_Pt3.

[0167] Specifically, in step S22, the three-dimensional coordinates of the frame thumb corner points are predicted and corrected by a Kalman filter algorithm to obtain the corrected three-dimensional coordinates of the frame thumb corner points, including the following operation steps:

[0168] Step S221: establishing a motion model for the three-dimensional coordinates of the thumb corner points of each frame;

[0169] Step S222: traversing the three-dimensional coordinates of the thumb corner points of each frame, predicting the three-dimensional coordinates of the thumb corner points of the frame corresponding to the three-dimensional coordinates of the thumb corner points of the continuous frames and the motion model, and obtaining the predicted three-dimensional coordinates corresponding to the three-dimensional coordinates of the thumb corner points of the frame of the t-th frame;

[0170] Step S223: correcting the predicted three-dimensional coordinates based on the three-dimensional coordinates of the thumb corner point in the frame t-1 to obtain corrected three-dimensional coordinates of the thumb corner point;

[0171] Repeat the above operation until all the three-dimensional coordinates of the thumb corner points of the frames are traversed.

[0172] It should be noted that the above embodiment of the present application corrects the four spatial position points respectively by designing a Kalman filter algorithm;

[0173] First, a motion model is established for each key point, and the position information of the historical frame is saved (the position information of the historical frame is the three-dimensional coordinates of the thumb corner point of the frame corresponding to the coordinates of the thumb corner point under the above continuous frames);

[0174] Take one of the points Left_Thumb_Pt0 as an example, use the motion model and historical frame information of this key point (i.e. the three-dimensional coordinates of the thumb corner point of the frame) to predict the position state of the t-th frame (the position state of the t-th frame is the predicted three-dimensional coordinates of the t-th frame) Left_Thumb_Pt0_pred_hat, and use the spatial position of the key point of the t-th frame obtained by the previous triangulation (i.e. the three-dimensional coordinates of the thumb corner point of the frame t-1, the t-1 frame is the previous frame, if under some special circumstances there is no three-dimensional coordinates of the thumb corner point of the previous frame, then the three-dimensional coordinates of the thumb corner point of the frame t is directly used as the modified three-dimensional coordinates of the thumb corner point) Left_Thumb_Pt0 as the observation value, update the predicted state to obtain the corrected position Left_Thumb_Pt0_hat;

[0175] Similarly, the corrected key point positions are obtained (the corrected key point positions are the corrected frame thumb corner three-dimensional coordinates of the t-th frame): Left_Thumb_Pt1_hat, Left_Thumb_Pt2_hat, Left_Thumb_Pt3_hat.

[0176] Specifically, in step S23, based on the initial thumb corner three-dimensional coordinates and the corrected frame thumb corner three-dimensional coordinates, the thumb posture corresponding to each corrected frame thumb corner three-dimensional coordinate is calculated by the ICP algorithm, including the following operation steps:

[0177] Step S231: calculating and obtaining the initial centroid coordinates of each of the initial thumb corner point three-dimensional coordinates and the frame centroid coordinates of each of the corrected frame thumb corner point three-dimensional coordinates;

[0178] Step S232: Calculate the initial relative centroid coordinates based on the initial centroid coordinates and the three-dimensional coordinates of each initial thumb corner point; and calculate the frame relative centroid coordinates based on the corrected three-dimensional coordinates of the thumb corner point and the frame centroid coordinates;

[0179] The calculation method of the initial relative centroid coordinates is: initial relative centroid coordinates = initial thumb corner three-dimensional coordinates - initial centroid coordinates;

[0180] The frame relative centroid coordinates are calculated as follows: frame relative centroid coordinates = corrected thumb corner three-dimensional coordinates - frame centroid coordinates;

[0181] Step S233: performing normalization operation on each of the initial relative centroid coordinates to obtain an initial normalized point set; performing normalization operation on each of the frame relative centroid coordinates to obtain a frame normalized point set;

[0182] Step S234: Calculating the thumb posture (R, T) corresponding to each corrected frame thumb corner point three-dimensional coordinate by the least square method based on the initial normalized point set and the frame normalized point set;

[0183] The thumb posture (R, T) is calculated as follows:

[0184] Where R is the rotation matrix; T is the translation vector; argmin is the input value corresponding to the minimum value of the function; sp_i is the i-th initial relative centroid coordinate in the initial normalized point set; dp_i is the i-th frame relative centroid coordinate in the frame normalized point set;

[0185] It should be noted that the above embodiment of the present application calculates the thumb posture of the t-th frame by using the ICP (Iterative Closest Point) algorithm;

[0186] In the above embodiment of the present application, the initial three-dimensional spatial positions of the thumb Left_Thumb_Init_Pt0, Left_Thumb_Init_Pt1, Left_Thumb_Init_Pt2, Left_Thumb_Init_Pt3, and the three-dimensional spatial positions of the thumb Left_Thumb_Pt0_hat, Left_Thumb_Pt1_hat, Left_Thumb_Pt2_hat, Left_Thumb_Pt3_hat of the tth frame have been obtained;

[0187] First, the centroid coordinates of each group of three-dimensional position points are subtracted respectively to obtain the normalized results, namely, the initial normalized point set Src_Pts = {sp_0, sp_1, sp_2, sp_3} and the frame normalized point set Dst_Pts = {dp_0, dp_1, dp_2, dp_3}. Then, the least squares method is used to calculate the rotation matrix R and the translation vector T for each corrected frame thumb corner point three-dimensional coordinate, so that the positions of the two groups of three-dimensional points after transformation are most consistent, that is, the thumb posture corresponding to the above-mentioned corrected frame thumb corner point three-dimensional coordinates is obtained.

[0188] Specifically, in step S30, the directional size ratio of the thumb projected onto the mobile phone screen is calculated based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points, including the following operation steps:

[0189] Step S31: vertically projecting the spatial coordinates of each thumb corner point to the spatial intersection point in the target a priori plane;

[0190] The spatial intersection points represent the intersection points of the spatial coordinates of each thumb corner point projected onto the target a priori plane and the target a priori plane;

[0191] Step S32: Calculate and obtain the left Euclidean distance dist_x of each of the spatial intersection points perpendicular to the left edge of the target a priori plane and the upper Euclidean distance dist_y of each of the spatial intersection points perpendicular to the upper edge of the target a priori plane;

[0192] Step S33: Calculating the length L of the target a priori plane and the width W of the target a priori plane based on the spatial coordinates of each of the mobile phone intersection points;

[0193] Step S34: Calculate the ratio r_X in the X direction based on the left Euclidean distance dist_x and the length L of the target a priori plane; calculate the ratio r_Y in the Y direction based on the upper Euclidean distance dist_y and the width W of the target a priori plane;

[0194] The calculation method of the ratio r_X in the X direction is: r_X=dist_x÷L;

[0195] The calculation method of the ratio r_Y in the Y direction is: r_Y=dist_y÷W;

[0196] It should be noted that, in the above embodiment of the present application, firstly, the four key points of the thumb (ie, the spatial coordinates of the thumb corner points, or the corrected three-dimensional coordinates of the thumb corner points in the frame) are:

[0197] Left_Thumb_Pt0_hat, Left_Thumb_Pt1_hat, Left_Thumb_Pt2_hat, and Left_Thumb_Pt3_hat are respectively projected vertically to the spatial mobile phone plane (i.e., the target prior plane mentioned above), and the spatial intersection points with the plane are calculated to obtain Left_Thumb_Pt0_proj, Left_Thumb_Pt1_proj, Left_Thumb_Pt2_proj, and Left_Thumb_Pt3_proj.

[0198] Next, calculate the Euclidean distance dist_x and the Euclidean distance dist_y of each spatial intersection point perpendicular to the left edge of the mobile phone plane, as follows: Figure 4 As shown; at the same time, the length L (length) and width W (width) of the mobile phone can be calculated from the four projection points of the mobile phone (the projection points are the above-mentioned spatial intersections) Phone_Pt1_proj, Phone_Pt2_proj, Phone_Pt3_proj, and Phone_Pt4_proj. Therefore, the proportion r_X in the X direction and r_Y in the Y direction within the range of the mobile phone can be calculated for these four thumb key points.

[0199] That is, the directional size ratio of the thumb projected onto the mobile phone screen, including the ratio r_X in the X direction and the ratio r_Y in the Y direction, can be obtained and outputted; then the above-mentioned operation steps S31 to S34 are terminated.

[0200] Embodiment 2

[0201] like Figure 5 As shown, accordingly, the present invention also proposes a response mapping device for AR glasses in cooperation with a gesture operation terminal, comprising a collection and calculation unit 10, a calculation unit 20 and a fusion module 30;

[0202] Wherein, the acquisition and calculation unit 10 includes a binocular vision module 11 and a calculation module 12;

[0203] The binocular vision module 11 is used to obtain a binocular view mobile phone image containing a complete mobile phone outline;

[0204] The calculation module 12 is used to obtain the coordinates of the mobile phone corner points after corner point detection of the collected binocular view mobile phone image, and then use the coordinates of the mobile phone corner points to obtain the spatial coordinates of the mobile phone corner points by triangulation and plane fitting;

[0205] The binocular view mobile phone image includes a left-view mobile phone image and a right-view mobile phone image;

[0206] The binocular vision module 11 is also used to obtain a thumb image containing a thumb picture;

[0207] The calculation module 12 is further used to obtain the coordinates of the thumb corner points after corner point detection of the collected thumb image, and then use the coordinates of the thumb corner points to process the coordinates of the thumb corner points through triangulation, Kalman filtering algorithm and ICP algorithm in sequence to obtain the spatial coordinates of the thumb corner points and the thumb posture;

[0208] The thumb image includes a left-eye view thumb image and a right-eye view thumb image;

[0209] The calculation unit 20 is used to calculate the directional size ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points;

[0210] The directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y;

[0211] The fusion module 30 is used to transmit the binocular view mobile phone image to the AR device to obtain an AR display image; based on the direction size ratio and the thumb posture, the thumb image is fused with the AR display image to obtain a fused image.

[0212] In summary, the response mapping method and device of an AR glasses collaborative gesture operation terminal proposed in the example of the present invention first determine the accurate position of the mobile phone screen in three-dimensional space through the spatial coordinates of the mobile phone corner points, thereby providing a position basis for the mobile phone screen for subsequent steps; obtain an image containing the thumb, obtain the two-dimensional corner point coordinates of the thumb through corner point detection, and then process these two-dimensional corner point coordinates through triangulation, Kalman filtering and ICP algorithm to obtain the three-dimensional spatial coordinates and posture of the thumb, accurately track the position and posture of the thumb, and provide data support for fusing and displaying the actual position of the thumb; based on the spatial coordinates of the mobile phone corner points and the spatial coordinates of the thumb corner points, calculate the projection size ratio of the thumb on the mobile phone screen, convert the spatial coordinates of the thumb into the relative size on the mobile phone screen, and ensure that the relative position of the thumb is accurately displayed on the AR device; transmit the binocular view mobile phone image to the AR device, and based on the direction size ratio and the thumb posture, fuse the thumb image with the AR display image, and finally generate a fused image, and then display the synthesized image on the AR device, so as to realize the synchronous mapping of the mobile phone screen and the thumb image, and provide an augmented reality experience of user interaction;

[0213] Through deep learning technology and visual geometry algorithms, the spatial position and posture of the mobile terminal and gestures are calculated in real time, and then the spatial position of the gestures is projected onto the plane of the mobile terminal through a projection algorithm to obtain the position of the hand relative to the mobile terminal. This position information is uploaded to the AR device together with the mobile phone screen image, so that the AR device can display the position of the user's fingers while displaying the mobile phone screen, providing visual reference and help for the user's operation.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace part or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A response mapping method for AR glasses in collaboration with a gesture operation terminal, characterized in that: The steps are as follows: Acquire a binocular view mobile phone image containing a complete mobile phone outline, obtain the coordinates of the mobile phone corner points after corner point detection on the acquired binocular view mobile phone image, and then use the mobile phone corner point coordinates to obtain the spatial coordinates of the mobile phone corner points by triangulation and plane fitting; A thumb image containing a thumb picture is obtained, and the coordinates of the thumb corner points are obtained after corner point detection of the collected thumb image. Then, the coordinates of the thumb corner points are processed by triangulation, Kalman filtering algorithm and ICP algorithm in sequence to obtain the spatial coordinates of the thumb corner points and the thumb posture; The directional dimension ratio of the thumb projected onto the mobile phone screen is calculated based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points; The directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y; The binocular view mobile phone image is transmitted to the AR device to obtain an AR display image; the thumb image is fused with the AR display image based on the direction size ratio and the thumb posture to obtain a fused image.

2. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 1, characterized in that: The method of obtaining the mobile phone corner point coordinates by corner point detection of the collected binocular view mobile phone image, and then obtaining the mobile phone corner point spatial coordinates by triangulation and plane fitting using the mobile phone corner point coordinates, includes the following steps: Performing corner point detection on the binocular view mobile phone image through a pre-trained key point detection network model to obtain the mobile phone corner point coordinates corresponding to the binocular view mobile phone image; The coordinates of the corner points of the mobile phone are calculated by a triangulation algorithm using the internal and external parameters of the camera to obtain the three-dimensional coordinates corresponding to the coordinates of the corner points of the mobile phone; Perform plane fitting on the three-dimensional coordinates to obtain the spatial coordinates of the corner points of the mobile phone.

3. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 2, characterized in that: The pre-trained key point detection network model is trained based on the YOLO model; the acquisition of the pre-trained key point detection network model includes the following steps: Crawling through a web crawler method to obtain a plurality of sample mobile phone outline images; performing a preprocessing operation on the sample mobile phone outline images to obtain preprocessed sample mobile phone outline images; The pre-processing operation includes a picture size adjustment operation; Annotating the preprocessed sample mobile phone contour image to obtain a sample data set; Establishing an initial key point detection network model; training the initial key point detection network model based on the sample data set, and outputting a trained key point detection network model after the initial key point detection network model converges.

4. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 3, characterized in that: The three-dimensional coordinates are plane-fitted to obtain the spatial coordinates of the mobile phone corner points; specifically, the following steps are included: Calculate the average point coordinates P_m (x_m, y_m, z_m) based on each of the three-dimensional coordinates; Constructing a priori plane based on the average point coordinates P_m(x_m, y_m, z_m); An error matrix H is constructed based on the error terms from each of the three-dimensional coordinates to the priori plane; Performing a singular value decomposition operation on the error matrix H to obtain a decomposition matrix H'; Obtain the last column of eigenvectors of the right singular vector matrix as plane parameter A, plane parameter B, and plane parameter C; Calculate the plane parameter D according to the three-dimensional coordinates and the priori plane; Acquire a target priori plane according to the plane parameter A, the plane parameter B, the plane parameter C and the plane parameter D; Each of the three-dimensional coordinates is vertically projected onto the target a priori plane to obtain the spatial coordinates of the mobile phone corner points corresponding to each of the three-dimensional coordinates.

5. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 4, characterized in that: The method of using the thumb corner point coordinates to process the thumb corner point spatial coordinates and thumb posture through triangulation, Kalman filter algorithm and ICP algorithm in sequence includes the following steps: Obtaining camera internal and external parameters; calculating initial thumb corner point three-dimensional coordinates through triangulation method according to the camera internal and external parameters and the thumb corner point coordinates; Acquire the frame thumb corner coordinates of continuous frames; calculate the frame thumb corner coordinates by triangulation method to obtain the frame thumb corner three-dimensional coordinates corresponding to each frame; predict and correct the frame thumb corner three-dimensional coordinates by Kalman filtering algorithm to obtain the corrected frame thumb corner three-dimensional coordinates; Based on the initial three-dimensional coordinates of the thumb corner points and the corrected three-dimensional coordinates of the frame thumb corner points, the thumb postures corresponding to the corrected three-dimensional coordinates of the frame thumb corner points are calculated by the ICP algorithm; The thumb gesture includes a rotation matrix R and a translation vector T.

6. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 5, characterized in that: The method of calculating the initial three-dimensional coordinates of the thumb corner points by triangulation according to the camera internal and external parameters and the thumb corner point coordinates includes the following steps: Obtain the first thumb corner point coordinates of the thumb corner point in N frames when the mobile phone screen surface is in a stationary state; The first thumb corner point coordinates are calculated by triangulation according to the internal and external parameters of the camera to obtain the first thumb corner point three-dimensional coordinates; the average coordinates of the first thumb corner point three-dimensional coordinates corresponding to each of the thumb corner point coordinates are calculated; and the average coordinates are determined as the initial thumb corner point three-dimensional coordinates.

7. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 6, characterized in that: The predicting and correcting operation of the frame thumb corner point three-dimensional coordinates by using a Kalman filter algorithm to obtain the corrected frame thumb corner point three-dimensional coordinates includes the following steps: Establishing a motion model for the three-dimensional coordinates of the thumb corner points in each frame; Traversing the three-dimensional coordinates of the thumb corner points of each frame, predicting the three-dimensional coordinates of the thumb corner points of the frame corresponding to the three-dimensional coordinates of the thumb corner points of the continuous frames and the motion model, and obtaining the predicted three-dimensional coordinates corresponding to the three-dimensional coordinates of the thumb corner points of the frame of the t-th frame; Correcting the predicted three-dimensional coordinates based on the three-dimensional coordinates of the thumb corner point in frame t-1 to obtain corrected three-dimensional coordinates of the thumb corner point; Repeat the above operation until all the three-dimensional coordinates of the thumb corner points of the frames are traversed.

8. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 7, characterized in that: The method of calculating the thumb postures corresponding to the corrected frame thumb corner point three-dimensional coordinates based on the initial thumb corner point three-dimensional coordinates and the corrected frame thumb corner point three-dimensional coordinates by using the ICP algorithm comprises the following steps: Calculate and obtain the initial centroid coordinates of each of the initial thumb corner point three-dimensional coordinates and the frame centroid coordinates of each of the corrected frame thumb corner point three-dimensional coordinates; The initial relative centroid coordinates are calculated based on the initial centroid coordinates and the three-dimensional coordinates of each initial thumb corner point; and the frame relative centroid coordinates are calculated based on the corrected three-dimensional coordinates of the thumb corner point and the frame centroid coordinates; The initial relative centroid coordinates are calculated as follows: Initial relative centroid coordinates = initial thumb corner three-dimensional coordinates - initial centroid coordinates; The frame relative centroid coordinates are calculated as follows: Frame relative centroid coordinates = corrected thumb corner three-dimensional coordinates - frame centroid coordinates; Normalizing each of the initial relative centroid coordinates to obtain an initial normalized point set; normalizing each of the frame relative centroid coordinates to obtain a frame normalized point set; Based on the initial normalized point set and the frame normalized point set, the thumb posture (R, T) corresponding to each corrected frame thumb corner point three-dimensional coordinate is calculated by the least square method; The thumb posture (R, T) is calculated as follows: Where R is the rotation matrix; T is the translation vector; argmin is the input value corresponding to the minimum value of the function; sp_i is the i-th initial relative centroid coordinate in the initial normalized point set; dp_i is the i-th frame relative centroid coordinate in the frame normalized point set.

9. The response mapping method of an AR glasses cooperative gesture operation terminal according to claim 8, characterized in that: The step of calculating the directional dimension ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points comprises the following steps: The spatial coordinates of each thumb corner point are respectively vertically projected onto the spatial intersection points in the target a priori plane; The spatial intersection points represent the intersection points of the spatial coordinates of each thumb corner point projected onto the target a priori plane and the target a priori plane; Calculate and obtain the left Euclidean distance dist_x of each of the spatial intersection points perpendicular to the left edge of the target a priori plane and the upper Euclidean distance dist_y of each of the spatial intersection points perpendicular to the upper edge of the target a priori plane; Calculating the length L of the target a priori plane and the width W of the target a priori plane based on the spatial coordinates of each of the mobile phone intersections; The ratio r_X in the X direction is calculated based on the left Euclidean distance dist_x and the length L of the target prior plane; the ratio r_Y in the Y direction is calculated based on the upper Euclidean distance dist_y and the width W of the target prior plane.

10. A response mapping device for AR glasses in collaboration with a gesture operation terminal, characterized in that: It includes an acquisition and calculation unit, a calculation unit and a fusion module; Wherein, the acquisition and calculation unit includes a binocular vision module and a calculation module; The binocular vision module is used to obtain a binocular view mobile phone image containing a complete mobile phone outline; The calculation module is used to obtain the coordinates of the corner points of the mobile phone after corner point detection of the collected binocular view mobile phone image, and then use the coordinates of the mobile phone corner points to obtain the spatial coordinates of the mobile phone corner points by triangulation and plane fitting; The binocular view mobile phone image includes a left-view mobile phone image and a right-view mobile phone image; The binocular vision module is also used to obtain a thumb image containing a thumb picture; The calculation module is further used to obtain the coordinates of the thumb corner points after corner point detection of the collected thumb image, and then use the coordinates of the thumb corner points to process through triangulation, Kalman filter algorithm and ICP algorithm in sequence to obtain the spatial coordinates of the thumb corner points and the thumb posture; The thumb image includes a left-eye view thumb image and a right-eye view thumb image; The calculation unit is used to calculate the directional size ratio of the thumb projected onto the mobile phone screen based on the spatial coordinates of the thumb corner points and the spatial coordinates of the mobile phone corner points; The directional dimension ratio includes an X-direction dimension ratio r_X and a Y-direction dimension ratio r_Y; the fusion module is used to transmit the binocular view mobile phone image to the AR device to obtain an AR display image; based on the directional dimension ratio and the thumb gesture, the thumb image is fused with the AR display image to obtain a fused image.

Images