Gesture interaction method, system, readable storage medium and computer

Abstract

The present invention provides a gesture interaction method, system, readable storage medium and computer, the method comprising: obtaining a user's line of sight focus area for a virtual marker; inputting a hand image into a convolutional neural network model to perform convolution operation to obtain a convolution feature image; performing pixel processing on a hand encoding image based on the hand image to obtain a corresponding pixel feature image; importing a multi-scale fusion unit into a multi-resolution network model to perform model optimization to obtain a network optimization model; inputting the convolution feature image and the pixel feature image into the network optimization model to output a user's hand posture estimation result, and performing collision judgment based on the hand posture estimation result and the line of sight focus area to realize the user's gesture interaction. The present invention uses a network optimization model to perform posture estimation on the convolution feature image and the pixel feature image, reduces the influence of the network parameter amount on the hand posture estimation, and thus reduces the detection error of the hand joint points.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a gesture interaction method, system, readable storage medium and computer. Background Art

[0002] With the rapid development of science and technology and the improvement of people's living standards, more and more people have begun to rely on computer devices, thus giving rise to human-computer interaction methods.

[0003] Gesture interaction, as a typical interaction mode of human-computer interaction, can reduce the limitations caused by national geographical and cultural differences, and thus convey emotional information intuitively and accurately. Existing gesture interaction methods usually adopt two methods. One is that users wear wearable sensor devices and operate based on the gesture interaction technology of the device and the guidance of professional technicians, which leads to high maintenance and maintenance costs of the equipment during use, resulting in high human and physical costs; the second is to learn gesture images through machine learning algorithms, and use template matching to obtain target gestures and template gestures for matching analysis. However, with the continuous increase in data volume, the workload of this method is also increasing, resulting in bottlenecks in software development and a significant increase in R&D costs. Summary of the invention

[0004] Based on this, an object of the present invention is to provide a gesture interaction method, system, readable storage medium and computer to at least solve the deficiencies in the above-mentioned technology.

[0005] The present invention provides a gesture interaction method, comprising:

[0006] Based on the virtual marker presented by the 3D display device, obtaining a user's sight focus area on the virtual marker;

[0007] Collecting a hand image of the user, and inputting the hand image into a convolutional neural network model to perform a convolution operation to obtain a corresponding convolution feature image;

[0008] Coordinate encoding is performed on the hand image to obtain a hand coded image, and pixel processing is performed based on the hand coded image to obtain a corresponding pixel feature image;

[0009] Constructing a multi-resolution network model, and importing a multi-scale fusion unit into the multi-resolution network model for model optimization to obtain a network optimization model;

[0010] The convolution feature image and the pixel feature image are input into the network optimization model to output the hand posture estimation result of the user, and collision judgment is performed based on the hand posture estimation result and the line of sight focus area to realize the gesture interaction of the user.

[0011] Furthermore, based on the virtual marker presented by the 3D display device, the step of obtaining the user's sight focus area for the virtual marker includes:

[0012] Acquire a user image of the user, and crop a face region of the user image to obtain a corresponding face image;

[0013] Constructing a facial feature point detection model, and using the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points;

[0014] Eye feature points are selected from among the facial feature points, and the sight focus area of ​​the user is determined according to the eye feature points.

[0015] Furthermore, the step of collecting the hand image of the user and inputting the hand image into a convolutional neural network model for convolution operation to obtain a corresponding convolution feature image includes:

[0016] Marking joint points in the hand image according to hand joint features, and generating a corresponding heat map for each joint point;

[0017] Each of the heat maps is input into a convolutional neural network model for convolution operation to reduce the resolution of the heat map to obtain a corresponding convolution feature image.

[0018] Furthermore, the steps of performing coordinate encoding on the hand image to obtain a hand coded image, and performing pixel processing based on the hand coded image to obtain a corresponding pixel feature image include:

[0019] Performing coordinate transformation on the hand image to obtain corresponding non-quantized coordinates, and generating a corresponding feature map according to the non-quantized coordinates;

[0020] The feature map is pixel processed to calculate feature information of all relevant nodes in the hand image, an undirected graph between all relevant nodes in the hand image is defined, and information processing is performed based on the undirected graph and the feature information to obtain a corresponding pixel feature image.

[0021] Furthermore, the steps of constructing a multi-resolution network model include:

[0022] Constructing a network residual structure, and replacing the convolution algorithm in the network residual structure with a lightweight algorithm to obtain a lightweight network residual structure;

[0023] Optimize the hand posture feature learning algorithm in the lightweight network residual structure to obtain an optimized lightweight network residual structure, and use the optimized lightweight network residual structure to construct a corresponding multi-resolution network model.

[0024] The present invention also provides a gesture interaction system, comprising:

[0025] An area acquisition module, used to acquire a user's sight focus area on a virtual marker based on the virtual marker presented by the 3D display device;

[0026] A convolution processing module, used for collecting the hand image of the user, and inputting the hand image into a convolutional neural network model for performing a convolution operation to obtain a corresponding convolution feature image;

[0027] A feature processing module, used for performing coordinate encoding on the hand image to obtain a hand coded image, and performing pixel processing based on the hand coded image to obtain a corresponding pixel feature image;

[0028] A model optimization module is used to construct a multi-resolution network model and import a multi-scale fusion unit into the multi-resolution network model for model optimization to obtain a network optimization model;

[0029] A gesture interaction module is used to input the convolution feature image and the pixel feature image into the network optimization model to output the hand posture estimation result of the user, and perform collision judgment with the line of sight focus area based on the hand posture estimation result to realize the gesture interaction of the user.

[0030] Furthermore, the region acquisition module includes:

[0031] An image acquisition unit, used to acquire a user image of the user, and perform face region cropping on the user image to obtain a corresponding face image;

[0032] A feature point detection unit, used to construct a facial feature point detection model, and use the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points;

[0033] The feature point screening unit is used to screen out eye feature points from the facial feature points, and determine the user's sight focus area based on the eye feature points.

[0034] Furthermore, the convolution processing module includes:

[0035] A joint point marking unit, used to mark the joint points in the hand image according to the hand joint features, and generate a corresponding heat map for each joint point;

[0036] A convolution processing unit is used to input each of the heat maps into a convolutional neural network model for performing a convolution operation so as to reduce the resolution of the heat map to obtain a corresponding convolution feature image.

[0037] Furthermore, the feature processing module includes:

[0038] A coordinate conversion unit, used for performing coordinate conversion on the hand image to obtain corresponding non-quantized coordinates, and generating a corresponding feature map according to the non-quantized coordinates;

[0039] A feature processing unit is used to perform pixel processing on the feature map to calculate feature information of all relevant nodes in the hand image, define an undirected graph between all relevant nodes in the hand image, and perform information processing based on the undirected graph and the feature information to obtain a corresponding pixel feature image.

[0040] Furthermore, the model optimization module includes:

[0041] A residual structure construction unit, used to construct a network residual structure, and replace the convolution algorithm in the network residual structure with a lightweight algorithm to obtain a lightweight network residual structure;

[0042] The model construction unit is used to optimize the hand posture feature learning algorithm in the lightweight network residual structure to obtain an optimized lightweight network residual structure, and use the optimized lightweight network residual structure to construct a corresponding multi-resolution network model.

[0043] The present invention also provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned gesture interaction method is implemented.

[0044] The present invention further provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned gesture interaction method when executing the computer program.

[0045] The gesture interaction method, system, readable storage medium and computer of the present invention collect the user's hand image and use the convolution neural network to perform convolution operation on the hand image to obtain a convolution feature image, so as to enhance the expressive ability of hand features and improve the accuracy of hand posture estimation; pixel processing is performed based on the hand coding image to obtain a pixel feature image, and the joint coordinates are used to alleviate the coding error in the hand image, and the feature information of the hand joint points is further enhanced. A multi-resolution network model is constructed, and the multi-resolution network model is optimized to obtain a network optimization model. The network optimization model is used to perform posture estimation on the convolution feature image and the pixel feature image, and the influence of the network parameter amount on the hand posture estimation is reduced, thereby reducing the detection error of the hand joint points. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of a gesture interaction method in a first embodiment of the present invention;

[0047] Figure 2 for Figure 1 Detailed flow chart of step S101;

[0048] Figure 3 for Figure 1 Detailed flow chart of step S102;

[0049] Figure 4 for Figure 1 Detailed flow chart of step S103;

[0050] Figure 5 for Figure 1 Detailed flow chart of step S104;

[0051] Figure 6 is a structural block diagram of a gesture interaction system in a second embodiment of the present invention;

[0052] Figure 7 FIG. 4 is a structural block diagram of a computer in a third embodiment of the present invention.

[0053] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0054] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0056] Embodiment 1

[0057] See also Figure 1 , which shows a gesture interaction method in a first embodiment of the present invention, and the method specifically includes steps S101 to S105:

[0058] S101, based on a virtual marker presented by a 3D display device, obtaining a visual focus area of ​​a user on the virtual marker;

[0059] For further information, see Figure 2 , the step S101 specifically includes steps S1011 to S1013:

[0060] S1011, obtaining a user image of the user, and cropping a face region of the user image to obtain a corresponding face image;

[0061] S1012, constructing a facial feature point detection model, and using the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points;

[0062] S1013, filter out eye feature points from the facial feature points, and determine the sight focus area of ​​the user according to the eye feature points.

[0063] In a specific implementation, image information is obtained when a user observes a virtual marker presented by a 3D display device with naked eyes, the image information at least including image information of a human face, and a non-maximum suppression algorithm is used to perform non-maximum suppression on the above image information to exclude other regions outside the human face region, so as to obtain a human face image including the human face region;

[0064] Specifically, a facial feature point detection model is constructed based on a preset facial feature point detection algorithm, and the detection model is used to perform feature point detection on the above-mentioned facial image. The feature extraction network in the detection model selects MobileNetV3, and the multi-scale connection layer in the detection model is used to improve the capture of feature points in the facial image. After obtaining the corresponding facial feature points, the eye feature points in the facial feature points are screened out, wherein the eye feature points include eyeball contour points and corresponding pupil feature points. The pupil feature points and eyeball contour points are used to solve the image data to obtain the user's line of sight focus area.

[0065] S102, collecting a hand image of the user, and inputting the hand image into a convolutional neural network model to perform a convolution operation to obtain a corresponding convolution feature image;

[0066] For further information, see Figure 3 , the step S102 specifically includes steps S1021~S1022:

[0067] S1021, marking joint points in the hand image according to hand joint features, and generating a corresponding heat map for each joint point;

[0068] S1022, input each of the heat maps into a convolutional neural network model for convolution operation, so as to reduce the resolution of the heat map to obtain a corresponding convolution feature image.

[0069] In the specific implementation, the user's hand image is collected by a preset image acquisition device, wherein the image acquisition device includes but is not limited to a mobile phone, a camera and other devices with image acquisition function. According to the hand joint characteristics of the human body, the collected hand image is divided into 20 joints, namely finger joints and wrist joints. , and generate a corresponding heat map for each joint point. The point with the highest confidence in each heat map is used as the position of the corresponding joint point. It indicates that, , Indicates The two-dimensional coordinates of the joint points are obtained, and the obtained heat map is input into the convolutional neural network for convolution operation to reduce its resolution by one quarter to obtain the corresponding convolution feature image.

[0070] S103, performing coordinate encoding on the hand image to obtain a hand coded image, and performing pixel processing based on the hand coded image to obtain a corresponding pixel feature image;

[0071] For further information, see Figure 4 , the step S103 specifically includes steps S1031~S1032:

[0072] S1031, performing coordinate transformation on the hand image to obtain corresponding non-quantized coordinates, and generating a corresponding feature map according to the non-quantized coordinates;

[0073] S1032, performing pixel processing on the feature map to calculate feature information of all relevant nodes in the hand image, defining an undirected graph between all relevant nodes in the hand image, and performing information processing based on the undirected graph and the feature information to obtain a corresponding pixel feature image.

[0074] In the specific implementation, based on the 2D coordinates of the hand image above , during data preprocessing, the coordinates are transformed to obtain the corresponding non-quantized coordinates:

[0075] ;

[0076] In the formula, Indicates the downsampling rate.

[0077] Specifically, the non-quantized coordinates obtained above are used to generate corresponding feature maps to reduce the quantization error of the quantized coordinates and improve the accuracy of the network model's learning of the hand joint point coordinates:

[0078] ;

[0079] In the formula, , represents any pixel position in the feature map, represents the variance, Represents an exponential function.

[0080] Furthermore, the feature map obtained above is pixel processed to generate feature information of all relevant nodes in the hand image. Specifically, the feature map is defined as , extract the height of the feature map ,width And the number of channels , calculate the weight from pixel to joint point:

[0081] ;

[0082] In the formula, represents the convolution transformation function, Represents spatial normalization.

[0083] Perform weighted averaging based on the features of the joint points:

[0084] ;

[0085] In the formula, Indicates The feature information of each joint point, Indicates The feature map of the joint points, represents the convolution transfer function, Indicates The weight matrix of the joint points, for an element of .

[0086] Furthermore, an undirected graph between all relevant nodes in the hand image is defined, the feature information of each joint point is input, the dependency relationship between the joint points is modeled using a graph convolutional neural network, and the matrix algorithm is used to operate on the features of all relevant nodes to obtain the corresponding pixel feature image.

[0087] S104, constructing a multi-resolution network model, and importing a multi-scale fusion unit into the multi-resolution network model to perform model optimization, so as to obtain a network optimization model;

[0088] For further information, see Figure 5 , the step S104 specifically includes steps S1041-S1042:

[0089] S1041, constructing a network residual structure, and using a lightweight algorithm to replace the convolution algorithm in the network residual structure to obtain a lightweight network residual structure;

[0090] S1042, optimizing the hand posture feature learning algorithm in the lightweight network residual structure to obtain an optimized lightweight network residual structure, and using the optimized lightweight network residual structure to construct a corresponding multi-resolution network model.

[0091] In the specific implementation, a network residual structure is constructed, and a lightweight algorithm (in this embodiment, the lightweight algorithm uses the GhostNet algorithm and the MobileNetV3 algorithm) is used to replace the convolution algorithm in the network residual structure to ensure the low cost of the network residual structure to obtain a lightweight network residual structure. The ECA-Net algorithm is used to optimize the hand posture feature learning algorithm of the lightweight network residual structure to obtain a more accurate position of the hand key points, and the optimized lightweight network residual structure is used to construct the corresponding multi-resolution network model.

[0092] Furthermore, the multi-scale fusion unit is introduced into the multi-resolution network model obtained above, and features are integrated from high to low. Downsampling uses 3x3 convolution with a step size of 2, and upsampling uses 1x1 convolution and interpolation, so as to obtain a network optimization model.

[0093] S105, inputting the convolution feature image and the pixel feature image into the network optimization model to output the hand posture estimation result of the user, and performing collision judgment with the line of sight focus area based on the hand posture estimation result to realize the gesture interaction of the user.

[0094] In a specific implementation, the convolution feature image and the pixel feature image obtained above are input into a network optimization model, the convolution feature image and the pixel feature image are normalized by using the network optimization model, the two images are used as inputs for three-dimensional hand posture estimation, the three-dimensional hand posture is estimated by using a posture prior network, the normalized image is transformed with the standard coordinates, and the transformed coordinates are mapped to the y-axis to obtain the corresponding rotation matrix;

[0095] Specifically, the 2D posture loss, distance loss and direction loss of the convolution feature image and the pixel feature image are calculated, the skeletal constraints between the joints are constructed according to the 2D posture loss, distance loss, direction loss and their corresponding hyperparameters, the hand posture estimation result of the user is output according to the skeletal constraints and the rotation matrix, and the hand posture estimation result is subjected to collision judgment with the previously obtained line of sight focus area to determine whether the user's hand collides with the virtual marker and whether to grasp the virtual marker, thereby realizing the user's gesture interaction.

[0096] In summary, the gesture interaction method in the above embodiments of the present invention collects the user's hand image and uses a convolutional neural network to perform a convolution operation on the hand image to obtain a convolution feature image, so as to enhance the expressive ability of hand features and improve the accuracy of hand posture estimation; pixel processing is performed based on the hand coding image to obtain a pixel feature image, and the joint coordinates are used to alleviate the coding error in the hand image, and the feature information of the hand joint points is further enhanced. A multi-resolution network model is constructed, and the multi-resolution network model is optimized to obtain a network optimization model. The network optimization model is used to perform posture estimation on the convolution feature image and the pixel feature image, and the influence of the network parameters on the hand posture estimation is reduced, thereby reducing the detection error of the hand joint points.

[0097] Embodiment 2

[0098] Another aspect of the present invention is to provide a gesture interaction system. Figure 6 , which shows a gesture interaction system in a second embodiment of the present invention, the system comprises:

[0099] The area acquisition module 11 is used to acquire the user's sight focus area for the virtual marker based on the virtual marker presented by the 3D display device;

[0100] Furthermore, the region acquisition module 11 includes:

[0101] An image acquisition unit, used to acquire a user image of the user, and perform face region cropping on the user image to obtain a corresponding face image;

[0102] A feature point detection unit, used to construct a facial feature point detection model, and use the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points;

[0103] The feature point screening unit is used to screen out eye feature points from the facial feature points, and determine the user's sight focus area based on the eye feature points.

[0104] A convolution processing module 12 is used to collect the hand image of the user and input the hand image into a convolutional neural network model to perform a convolution operation to obtain a corresponding convolution feature image;

[0105] Furthermore, the convolution processing module 12 includes:

[0106] A joint point marking unit, used to mark the joint points in the hand image according to the hand joint features, and generate a corresponding heat map for each joint point;

[0107] A convolution processing unit is used to input each of the heat maps into a convolutional neural network model for performing a convolution operation so as to reduce the resolution of the heat map to obtain a corresponding convolution feature image.

[0108] A feature processing module 13, configured to perform coordinate encoding on the hand image to obtain a hand encoding image, and perform pixel processing based on the hand encoding image to obtain a corresponding pixel feature image;

[0109] Furthermore, the feature processing module 13 includes:

[0110] A coordinate conversion unit, used for performing coordinate conversion on the hand image to obtain corresponding non-quantized coordinates, and generating a corresponding feature map according to the non-quantized coordinates;

[0111] A feature processing unit is used to perform pixel processing on the feature map to calculate feature information of all relevant nodes in the hand image, define an undirected graph between all relevant nodes in the hand image, and perform information processing based on the undirected graph and the feature information to obtain a corresponding pixel feature image.

[0112] A model optimization module 14 is used to construct a multi-resolution network model and import a multi-scale fusion unit into the multi-resolution network model to perform model optimization to obtain a network optimization model;

[0113] Furthermore, the model optimization module 14 includes:

[0114] A residual structure construction unit, used to construct a network residual structure, and replace the convolution algorithm in the network residual structure with a lightweight algorithm to obtain a lightweight network residual structure;

[0115] The model construction unit is used to optimize the hand posture feature learning algorithm in the lightweight network residual structure to obtain an optimized lightweight network residual structure, and use the optimized lightweight network residual structure to construct a corresponding multi-resolution network model.

[0116] The gesture interaction module 15 is used to input the convolution feature image and the pixel feature image into the network optimization model to output the hand posture estimation result of the user, and perform collision judgment with the line of sight focus area based on the hand posture estimation result to realize the gesture interaction of the user.

[0117] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.

[0118] The implementation principle and technical effects of the gesture interaction system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0119] Embodiment 3

[0120] The present invention also provides a computer, see Figure 7 , shown is a computer in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned gesture interaction method is implemented.

[0121] The memory 10 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as a hard disk of the computer. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Further, the memory 10 may also include both an internal storage unit of the computer and an external storage device. The memory 10 may be used not only to store application software and various types of data installed in the computer, but also to temporarily store data that has been output or is to be output.

[0122] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.

[0123] It should be pointed out that Figure 7 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0124] The embodiment of the present invention further provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the gesture interaction method as described above is implemented.

[0125] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0127] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0128] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A gesture interaction method, characterized in that: include: Based on the virtual marker presented by the 3D display device, obtaining a user's sight focus area on the virtual marker; The hand image of the user is collected, and the hand image is input into a convolutional neural network model for convolution operation to obtain a corresponding convolutional feature image, wherein the steps of collecting the hand image of the user, and inputting the hand image into a convolutional neural network model for convolution operation to obtain a corresponding convolutional feature image include: According to the hand joint features, the collected hand image of the user is divided into 20 joints, namely finger joints and wrist joints. , and generate a corresponding heat map for each joint point, where the point with the largest confidence in each heat map is used as the position of the corresponding joint point, and express, , Indicates The two-dimensional coordinates of the joint points; Inputting each of the heat maps into a convolutional neural network model for convolution operation, so as to reduce the resolution of the heat map by one quarter, so as to obtain a corresponding convolution feature image; Coordinate encoding is performed on the hand image to obtain a hand coded image, and pixel processing is performed based on the hand coded image to obtain a corresponding pixel feature image, wherein the steps of coordinate encoding is performed on the hand image to obtain a hand coded image, and pixel processing is performed based on the hand coded image to obtain a corresponding pixel feature image include: The 2D coordinates of the hand image , which is transformed during data preprocessing to obtain the corresponding non-quantized coordinates: ； In the formula, represents the downsampling rate; The non-quantized coordinates are used to generate corresponding feature maps to reduce the quantization error of the quantized coordinates and improve the accuracy of the network model's learning of the hand joint point coordinates: ； In the formula, , represents any pixel position in the feature map, represents the variance, represents the exponential function; The feature map is pixel processed to calculate the feature information of all relevant nodes in the hand image, wherein the feature map is defined as , extract the height of the feature map ,width And the number of channels , calculate the weight from pixel to joint point: ； In the formula, represents the convolution transformation function, represents spatial normalization; Perform weighted averaging based on the features of the joint points: ； In the formula, Indicates The feature information of each joint point, Indicates The feature map of the joint points, represents the convolution transfer function, Indicates The weight matrix of the joint points, for an element of An undirected graph between all the nodes in the hand image is defined, feature information of each joint point is input, a graph convolutional neural network is used to model the dependency relationship between the joint points, and a matrix algorithm is used to calculate the features of all the nodes to obtain a corresponding pixel feature image; Constructing a multi-resolution network model, and importing a multi-scale fusion unit into the multi-resolution network model for model optimization to obtain a network optimization model; The convolution feature image and the pixel feature image are input into the network optimization model to output the hand posture estimation result of the user, and collision judgment is performed based on the hand posture estimation result and the line of sight focus area to realize the gesture interaction of the user.

2. The gesture interaction method according to claim 1, characterized in that: Based on the virtual marker presented by the 3D display device, the step of obtaining the user's sight focus area for the virtual marker includes: Acquire a user image of the user, and crop a face region of the user image to obtain a corresponding face image; Constructing a facial feature point detection model, and using the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points; Eye feature points are selected from among the facial feature points, and the sight focus area of ​​the user is determined according to the eye feature points.

3. The gesture interaction method according to claim 1, characterized in that: The steps to build a multi-resolution network model include: Constructing a network residual structure, and replacing the convolution algorithm in the network residual structure with a lightweight algorithm to obtain a lightweight network residual structure; Optimize the hand posture feature learning algorithm in the lightweight network residual structure to obtain an optimized lightweight network residual structure, and use the optimized lightweight network residual structure to construct a corresponding multi-resolution network model.

4. A gesture interaction system, characterized in that: include: An area acquisition module, used to acquire a user's sight focus area on a virtual marker based on the virtual marker presented by the 3D display device; A convolution processing module is used to collect the hand image of the user and input the hand image into a convolutional neural network model to perform a convolution operation to obtain a corresponding convolution feature image, wherein the convolution processing module includes: The joint point marking unit is used to divide the collected hand image of the user into 20 joints according to the hand joint features, namely finger joints and wrist joints. , and generate a corresponding heat map for each joint point, where the point with the largest confidence in each heat map is used as the position of the corresponding joint point, and express, , Indicates The two-dimensional coordinates of the joint points; A convolution processing unit, used for inputting each of the heat maps into a convolutional neural network model to perform a convolution operation, so as to reduce the resolution of the heat map by one quarter, so as to obtain a corresponding convolution feature image; A feature processing module is used to perform coordinate encoding on the hand image to obtain a hand coded image, and perform pixel processing based on the hand coded image to obtain a corresponding pixel feature image, wherein the feature processing module includes: A coordinate conversion unit for converting the 2D coordinates of the hand image , which is transformed during data preprocessing to obtain the corresponding non-quantized coordinates: ； In the formula, represents the downsampling rate; The non-quantized coordinates are used to generate corresponding feature maps to reduce the quantization error of the quantized coordinates and improve the accuracy of the network model's learning of the hand joint point coordinates: ； In the formula, , represents any pixel position in the feature map, represents the variance, represents the exponential function; A feature processing unit is used to perform pixel processing on the feature map to calculate feature information of all relevant nodes in the hand image, wherein the feature map is defined as , extract the height of the feature map ,width And the number of channels , calculate the weight from pixel to joint point: ； In the formula, represents the convolution transformation function, represents spatial normalization; Perform weighted averaging based on the features of the joint points: ； In the formula, Indicates The feature information of each joint point, Indicates The feature map of the joint points, represents the convolution transfer function, Indicates The weight matrix of the joint points, for an element of An undirected graph between all the nodes in the hand image is defined, feature information of each joint point is input, a graph convolutional neural network is used to model the dependency relationship between the joint points, and a matrix algorithm is used to calculate the features of all the nodes to obtain a corresponding pixel feature image; A model optimization module is used to construct a multi-resolution network model and import a multi-scale fusion unit into the multi-resolution network model for model optimization to obtain a network optimization model; A gesture interaction module is used to input the convolution feature image and the pixel feature image into the network optimization model to output the hand posture estimation result of the user, and perform collision judgment with the line of sight focus area based on the hand posture estimation result to realize the gesture interaction of the user.

5. The gesture interaction system according to claim 4, characterized in that: The region acquisition module comprises: An image acquisition unit, used to acquire a user image of the user, and perform face region cropping on the user image to obtain a corresponding face image; A feature point detection unit, used to construct a facial feature point detection model, and use the facial feature point detection model to perform feature point detection on the facial image to obtain a plurality of facial feature points; The feature point screening unit is used to screen out eye feature points from the facial feature points, and determine the user's sight focus area based on the eye feature points.

6. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the gesture interaction method as described in any one of claims 1 to 3 is implemented.

7. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the gesture interaction method according to any one of claims 1 to 3 is implemented.

Images