A gesture recognition method and system for flying screen interaction

Abstract

The present application provides a gesture recognition method and system for flying screen interaction, which relate to the field of gesture recognition technology, including: using a background subtraction algorithm to process video data to obtain a foreground area in the video data; using human posture estimation to extract the foreground area in the video data to obtain a set of hand key points of each human body; using forward kinematics to construct a hand skeleton model based on the set of hand key points of each human body; using the hand skeleton model to simulate hand movement and extract the gesture area and actual spatial position of each human body; based on the gesture area and actual spatial position of each human body, by combining distance measurement and collision detection algorithm, determining whether spatial overlap and intersection occur in the gestures of different users, distinguishing between independent gestures and interactive gestures of each user, avoiding the misjudgment problem of conventional gesture recognition methods in the state of gesture overlap and intersection, and thereby improving the accuracy of gesture recognition.

Description

Technical Field

[0001] The present application relates to the field of gesture recognition technology, and in particular to a gesture recognition method and system for flying screen interaction. Background Art

[0002] In modern human-computer interaction, users can achieve cross-device operations and interactions through gesture recognition. However, in actual applications, in multi-user interaction scenarios, if multiple users use gestures to operate at the same time, the spatial positions of user gestures may overlap and intersect. Conventional gesture recognition methods focus on single-user scenarios and have insufficient stability when identifying whether gestures intersect and overlap. Due to the lack of stability, conventional gesture recognition methods are prone to misjudging gestures in overlapping and intersecting states, thereby reducing the accuracy of gesture recognition. Summary of the Invention

[0003] The present application provides a gesture recognition method and system for flying screen interaction, aiming to solve the problem of insufficient stability in the related art when identifying whether gestures are crossed and overlapped, thereby avoiding the problem of misjudging gestures in overlapping and crossing states.

[0004] To achieve the above objectives, this application provides a gesture recognition method for flying screen interaction, including the following:

[0005] Get video data of user gestures.

[0006] The background subtraction algorithm is used to process the video data to obtain the foreground area in the video data.

[0007] Human body posture estimation is used to extract the foreground area in the video data to obtain the set of key points of each human hand.

[0008] According to the set of key points of each human hand, the hand skeleton model is constructed using forward kinematics.

[0009] The hand skeleton model is used to simulate the movement of the hand and extract the gesture area and actual spatial position of each body.

[0010] Based on the gesture area and actual spatial position of each body, distance measurement combined with collision detection is used to determine whether the gestures of different users overlap or intersect in space.

[0011] Trigger interactive feedback based on the judgment results.

[0012] As a preferred solution of the present invention, a background subtraction algorithm is used to process the video data to obtain the foreground area in the video data, specifically including:

[0013] Initialize the Gaussian mixture model parameters; the Gaussian mixture model parameters include: the number of Gaussian distributions, the learning rate, and the foreground determination threshold.

[0014] Construct a Gaussian mixture model based on the initialized Gaussian mixture model parameters.

[0015] A plurality of video frames containing a human portrait are extracted from the video data, each of the video frames containing a plurality of pixel points.

[0016] A Gaussian mixture model is used to establish a background model corresponding to each pixel; the background model includes at least one Gaussian distribution.

[0017] The total matching degree of each pixel is calculated based on the Gaussian distribution in the background model corresponding to each pixel, and the current pixel is judged as the foreground area or the background area according to the total matching degree.

[0018] If the current pixel is considered to be the foreground area, the background model is updated to adapt to the current scene changes; if the current pixel is considered to be the background, the background model is not updated and the current background model is still used.

[0019] The pixels in the foreground area of ​​each video frame are marked as 1, and the pixels in the background area of ​​each video frame are marked as 0, thereby generating a foreground mask image of each video frame.

[0020] The foreground mask images in each video frame are merged to track and extract the foreground area in the video data.

[0021] As a preferred solution of the present invention, human posture estimation is used to extract the foreground area in the video data to obtain a set of hand key points of each human body, specifically including:

[0022] A human posture estimation model is used to identify the foreground area in the video data to obtain multiple limb key points related to the human body in the video data; the multiple limb key points include the head, torso, limbs and hands.

[0023] According to the fixed index of each hand corresponding to the human body in the key point sequence, the key point information related to the hand is output; the key point information related to the hand includes the two-dimensional coordinates of the wrist and finger joint positions.

[0024] A set of key points of the hand corresponding to each human body is generated according to key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

[0025] As a preferred solution of the present invention, a hand skeleton model is constructed using forward kinematics based on a set of hand key points of each human body, specifically including:

[0026] The connection relationship between joint nodes and joints is defined based on the set of key points of each human hand, and the spatial transformation relationship between each joint is calculated.

[0027] According to the spatial transformation relationship between each joint, a hand skeleton model is constructed, which includes the three-dimensional positions and relative motion relationships of all joints.

[0028] As a preferred solution of the present invention, a hand skeleton model is used to simulate hand movements and extract the gesture area and actual spatial position of each human body, specifically including:

[0029] A 3D bounding box is constructed based on all key 3D positions within the hand skeleton model; within the 3D bounding box, the gesture area is calculated based on the relative motion relationship within the hand skeleton model.

[0030] Based on the continuous frames in the video data, the changes in the hand skeleton model corresponding to each human body are tracked, and the motion trajectory of each joint corresponding to each human body in three-dimensional space is calculated.

[0031] The actual position of each human hand in space is extracted according to the motion trajectory of each joint corresponding to each human body in three-dimensional space.

[0032] As a preferred solution of the present invention, based on the gesture area and actual spatial position of each body, distance measurement combined with collision detection is used to determine whether the gestures of different users overlap or intersect in space, specifically including:

[0033] The center point distance between each human gesture is calculated based on the gesture area and actual spatial position of each human gesture.

[0034] Determine whether the center point distance is less than the overlap threshold; when the center point distance is less than the overlap threshold, it is determined that the two gesture areas have preliminary overlap, otherwise there is no preliminary overlap.

[0035] When there is a preliminary overlap between two gesture areas, the axis-aligned bounding box collision detection algorithm is used to determine whether the two bounding boxes intersect based on the spatial overlap condition. If there is an intersection, the two gesture areas are determined to have spatial overlap; if there is no intersection, the two gesture areas are determined to have no spatial overlap.

[0036] When two gesture areas overlap in space, the intersection volume of the two gesture areas is calculated to determine whether the intersection volume is greater than 0; if the intersection volume is greater than 0, it is determined that the two gesture areas have spatially intersected, otherwise it is considered that there is no spatial intersection.

[0037] As a preferred solution of the present invention, triggering interactive feedback according to the judgment result specifically includes:

[0038] When there is no spatial overlap between gestures, the center point distance is greater than the judgment threshold, or the bounding boxes have no intersection, and when there is no spatial overlap between gesture areas, each gesture area is considered independent, and the instructions of each gesture are processed independently without considering interference between users. Otherwise, the current state of the gesture is specifically judged and feedback is triggered.

[0039] This application also provides a gesture recognition system for flying screen interaction, including the following modules:

[0040] The data acquisition module is used to acquire video data of user gestures.

[0041] The background subtraction module is used to process the video data using a background subtraction algorithm to obtain the foreground area in the video data.

[0042] The data extraction module is used to extract the foreground area in the video data by using human posture estimation to obtain a set of key points of the hands of each human body.

[0043] The hand modeling module is used to construct a hand skeleton model using forward kinematics based on the set of key points of the hand of each human body.

[0044] The simulation recognition module is used to simulate the movement of the hand using the hand skeleton model and extract the gesture area and actual spatial position of each human body.

[0045] The comprehensive judgment module is used to determine whether the gestures of different users overlap or intersect in space based on the gesture area and actual spatial position of each body through distance measurement combined with collision detection.

[0046] The feedback module is used to trigger interactive feedback based on the judgment results.

[0047] As a preferred solution of the present invention, the background subtraction module specifically includes:

[0048] The initialization unit is used to initialize the parameters of the Gaussian mixture model; the Gaussian mixture model parameters include: the number of Gaussian distributions, the learning rate and the foreground determination threshold.

[0049] The Gaussian mixture model construction unit is used to construct a Gaussian mixture model according to the initialized Gaussian mixture model parameters.

[0050] The multiple video frame extraction units are used to extract multiple video frames containing human portraits from the video data, each of the video frames containing multiple pixel points.

[0051] The background model construction unit is used to establish a background model corresponding to each pixel point using a Gaussian mixture model; the background model includes at least one Gaussian distribution.

[0052] The judgment unit is used to calculate the total matching degree corresponding to each pixel point based on each Gaussian distribution in the background model corresponding to each pixel point, and judge whether the current pixel point is a foreground area or a background area according to the total matching degree.

[0053] The background model updating unit is used to update the background model to adapt to the current scene changes if the current pixel point is considered to be the foreground area; if the current pixel point is considered to be the background, the background model is not updated and the current background model is still used.

[0054] The foreground mask image generating unit is used to mark the pixels in the foreground area of ​​each video frame as 1 and mark the pixels in the background area of ​​each video frame as 0, so as to generate a foreground mask image of each video frame.

[0055] The foreground mask image merging unit is used to merge the foreground mask images in each video frame, track and extract the foreground area in the video data.

[0056] As a preferred solution of the present invention, the data extraction module specifically includes:

[0057] The limb key point recognition unit is used to identify the foreground area in the video data using a human posture estimation model to obtain multiple limb key points related to the human body in the video data; the multiple limb key points include the head, torso, limbs and hands.

[0058] The key point information determination unit is used to output key point information related to the hand according to the fixed index of the hand corresponding to each human body in the key point sequence; the key point information related to the hand includes the two-dimensional coordinates of the wrist and finger joint positions.

[0059] The hand key point set generating unit is used to generate a set of hand key points corresponding to each human body according to the key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

[0060] The beneficial effects of this application are:

[0061] 1. By combining distance measurement with collision detection algorithms, we can determine whether different users' gestures overlap or intersect in space, effectively distinguishing between independent gestures and interactive gestures. This avoids the misjudgment problem of conventional gesture recognition methods in the case of overlapping or intersecting gestures, thereby improving the accuracy of gesture recognition.

[0062] 2. Human body posture estimation and hand skeleton modeling technology are used to not only dynamically track the gesture area, but also simulate hand movements through forward kinematics. Combined with motion trajectory analysis of continuous frames, this alleviates interference problems caused by overlapping and intersecting gestures of multiple users, thereby improving gesture recognition stability.

[0063] 3. Dynamic background updating and processing are implemented based on the Gaussian mixture model to adapt to changing environmental conditions, reduce the impact of background noise, and improve processing efficiency, ensuring that gesture recognition remains stable in complex backgrounds.

[0064] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 A flowchart of a gesture recognition method for flying screen interaction provided in an embodiment of the present application.

[0067] Figure 2 This is a diagram of the overall architecture of a gesture recognition system for flying screen interaction provided in an embodiment of the present application.

[0068] Description of the accompanying drawings: 21. Data acquisition module; 22. Background subtraction module; 23. Data extraction module; 24. Hand modeling module; 25. Simulation recognition module; 26. Comprehensive judgment module; 27. Feedback module. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0070] See also Figure 1 , Figure 1 A flowchart of a gesture recognition method for flying screen interaction is provided for an embodiment of the present application.

[0071] In this embodiment, a gesture recognition method for flying screen interaction includes steps S100, S200, S300, S400, S500, S600 and S700.

[0072] Step S100, obtaining video data of user gestures, specifically includes:

[0073] Configure the video capture parameters of the video capture device, including resolution, frame rate, and color mode. After the configuration is complete, start the video capture device and obtain video data from the video capture device. The video data includes continuous video frames to reflect the dynamic changes of the user's gestures.

[0074] Among them, the video capture device includes a depth camera and a camera.

[0075] Step S200: Processing the video data using a background subtraction algorithm to obtain a foreground area in the video data, specifically including:

[0076] Initialize the Gaussian mixture model parameters; the Gaussian mixture model parameters include: the number of Gaussian distributions, the learning rate, and the foreground determination threshold.

[0077] It should be noted that the number of Gaussian distributions is determined through 5-fold cross-validation, and the initial learning rate is limited to between 0.01 and 0.1. The foreground judgment threshold is determined based on the matching degree of each Gaussian distribution in the Gaussian mixture model. The initial value is set to 0.5 and is dynamically adjusted according to the color distribution of the foreground and background during the subsequent judgment process.

[0078] Construct a Gaussian mixture model based on the initialized Gaussian mixture model parameters.

[0079] A plurality of video frames containing a human portrait are extracted from the video data, each of the video frames containing a plurality of pixel points.

[0080] A Gaussian mixture model is used to establish a background model corresponding to each pixel; the background model includes at least one Gaussian distribution.

[0081] The total matching degree of each pixel is calculated based on the Gaussian distribution in the background model corresponding to each pixel, and the current pixel is judged as the foreground area or the background area according to the total matching degree.

[0082] It should be noted that the total matching degree corresponding to each pixel is calculated based on the Gaussian distribution in the background model corresponding to each pixel. The calculation formula is:

[0083]

[0084] Where, ω k is the weight of the kth Gaussian distribution, which is determined by the contribution of the kth Gaussian distribution to the overall model; Is a Gaussian distribution function, which is used to express the matching degree of the k-th Gaussian distribution to the color value I(x, y), where μ k and σ k are the mean and standard deviation of the Gaussian distribution respectively; P(I(x, y)) is the total matching degree corresponding to the current pixel color value I(x, y), p(x, y) represents the current pixel, and I(x, y) represents the color value of the current pixel.

[0085] It should be further explained that the total matching degree is determined to be less than the foreground determination threshold. If the matching degree of the current pixel is less than the foreground determination threshold, the pixel is considered to belong to the foreground area; otherwise, it is considered to belong to the background area. The judgment process is as follows:

[0086] if P(I(x,y))<T foreground , then p(x, y) = foreground

[0087] else if P(I(x,y))≥T foreground , then p(x, y) = background

[0088] Where P(I(x, y)) represents the matching degree of pixel point (x, y) in the current video frame I(x, y); T foreground Represents the foreground judgment threshold, and p(x, y) indicates whether the pixel point (x, y) belongs to the foreground area.

[0089] If the current pixel is considered to be the foreground area, the background model is updated to adapt to the current scene changes; if the current pixel is considered to be the background, the background model is not updated and the current background model is still used.

[0090] The pixels in the foreground area of ​​each video frame are marked as 1, and the pixels in the background area of ​​each video frame are marked as 0, thereby generating a foreground mask image of each video frame.

[0091] The foreground mask images in each video frame are merged to track and extract the foreground area in the video data.

[0092] It should be noted that the foreground mask images obtained in each video frame are merged to generate a global foreground area map, and Kalman filtering is used as a tracking algorithm to extract and track the foreground area in the video frame. According to the global foreground area map and the tracking results, the foreground area in the video data is extracted.

[0093] Step S300: extracting the foreground area in the video data by using human posture estimation to obtain a set of hand key points of each human body, specifically including:

[0094] A human posture estimation model is used to identify the foreground area in the video data to obtain multiple limb key points related to the human body in the video data; the multiple limb key points include the head, torso, limbs and hands.

[0095] It should be noted that OpenPose in the deep learning human pose estimation model is used for limb recognition, multiple limb key points of the human body are detected in the foreground area, and the information of multiple limb key points is extracted through a convolutional neural network.

[0096] According to the fixed index of each hand corresponding to the human body in the key point sequence, the key point information related to the hand is output; the key point information related to the hand includes the two-dimensional coordinates of the wrist and finger joint positions.

[0097] It should be noted that in the key point sequence, the index of the hand key point is fixed, and the hand-related key points are directly located through the index.

[0098] A set of key points of the hand corresponding to each human body is generated according to key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

[0099] It should be noted that the specific collection is:

[0100] P hand ={(x i ,y i )|i=1,2,...,h}

[0101] Where, P hand is the set of key points of the hand corresponding to each human body, (x i ,y i ) represents the two-dimensional coordinates of each key point corresponding to each human body, and h is the number of key points on the hand.

[0102] Step S400: constructing a hand skeleton model using forward kinematics based on the hand key point set of each human body, specifically including:

[0103] The connection relationship between joint nodes and joints is defined based on the set of key points of each human hand, and the spatial transformation relationship between each joint is calculated.

[0104] It should be noted that, based on the set of key points of the hand, the various joint nodes of the hand are defined, the connection relationship between the joints is defined through anatomical knowledge, and then the spatial transformation relationship between the joints is calculated.

[0105] It should be further explained that the forward kinematics method is used to calculate the rotation angles and displacements between adjacent joints to obtain the spatial transformation relationship between the joints.

[0106] According to the spatial transformation relationship between each joint, a hand skeleton model is constructed, which includes the three-dimensional positions and relative motion relationships of all joints.

[0107] Step S500: simulating hand movements using the hand skeleton model to extract the gesture area and actual spatial position of each human body, specifically including:

[0108] A 3D bounding box is constructed based on all key 3D positions within the hand skeleton model; within the 3D bounding box, the gesture area is calculated based on the relative motion relationship within the hand skeleton model.

[0109] It should be noted that the size and shape of the 3D bounding box are updated with the dynamic changes of the hand. As the hand moves, the 3D bounding box will be adjusted in real time to enclose all key points of the hand model to ensure that the entire gesture area is covered.

[0110] Based on the continuous frames in the video data, the changes in the hand skeleton model corresponding to each human body are tracked, and the motion trajectory of each joint corresponding to each human body in three-dimensional space is calculated.

[0111] The actual position of each human hand in space is extracted according to the motion trajectory of each joint corresponding to each human body in three-dimensional space.

[0112] Step S600: Based on the gesture area and actual spatial position of each human body, determine whether the gestures of different users overlap or intersect in space by combining distance measurement with collision detection, specifically including:

[0113] The center point distance between each human gesture is calculated based on the gesture area and actual spatial position of each human gesture.

[0114] Determine whether the center point distance is less than the overlap threshold; when the center point distance is less than the overlap threshold, it is determined that the two gesture areas have preliminary overlap, otherwise there is no preliminary overlap.

[0115] When there is a preliminary overlap between two gesture areas, the axis-aligned bounding box collision detection algorithm is used to determine whether the two bounding boxes intersect based on the spatial overlap condition. If there is an intersection, the two gesture areas are determined to have spatial overlap; if there is no intersection, the two gesture areas are determined to have no spatial overlap.

[0116] It should be noted that the spatial overlap conditions are:

[0117]

[0118]

[0119] Where, Indicates the maximum coordinate of gesture area i on the x-axis; Indicates the minimum coordinate of gesture area i on the x-axis; Indicates the maximum coordinate of gesture area j on the x-axis; Indicates the minimum coordinate of gesture area j on the x-axis; Indicates the maximum coordinate of gesture area i on the y-axis; Indicates the minimum coordinate of gesture area i on the y-axis; Indicates the maximum coordinate of gesture area j on the y-axis; Indicates the minimum coordinate of gesture area j on the y-axis; Indicates the maximum coordinate of gesture area i on the z-axis; Indicates the minimum coordinate of gesture area i on the z-axis; Indicates the maximum coordinate of gesture area j on the z-axis; Indicates the minimum coordinate of gesture area j on the z-axis.

[0120] The purpose of the spatial overlap formula is to determine whether two 3D gesture areas intersect. The judgment logic is as follows:

[0121] For the overlap condition in the x-axis direction: For the x-axis, Indicates that the right side of gesture area i exceeds the left side of gesture area j; It means that the left side of gesture area i is located before the right side of gesture area j. If these two conditions are met, it means that there is overlap in the x-axis direction.

[0122] For the overlap condition in the y-axis direction: For the y-axis, and Represents the overlap in the y-axis direction.

[0123] For the overlap condition in the z-axis direction: For the z-axis, and Indicates overlap in the z-axis direction.

[0124] If the overlap conditions of the three axes are met, that is, the ranges of the x, y, and z axes intersect, the two gesture areas are considered to overlap in three-dimensional space, and further collision detection is performed.

[0125] When two gesture areas overlap in space, the intersection volume of the two gesture areas is calculated to determine whether the intersection volume is greater than 0; if the intersection volume is greater than 0, it is determined that the two gesture areas have spatially intersected, otherwise it is considered that there is no spatial intersection.

[0126] It should be noted that the calculation formula for the intersection volume of two gesture areas is:

[0127]

[0128] Where V intersection represents the intersection volume of the two gesture area bounding boxes, Indicates the maximum and minimum values ​​of the intersection area in the x-axis direction, Indicates the maximum and minimum values ​​of the intersection area in the y-axis direction, Indicates the maximum and minimum values ​​of the intersection area in the z-axis direction.

[0129] Step S700: triggering interactive feedback according to the judgment result, specifically including:

[0130] When there is no spatial overlap between gestures, the center point distance is greater than the judgment threshold, or the bounding boxes have no intersection, and when there is no spatial overlap between gesture areas, each gesture area is considered independent, and the instructions of each gesture are processed independently without considering interference between users. Otherwise, the current state of the gesture is specifically judged and feedback is triggered.

[0131] It should be noted that when the gestures overlap in space but do not intersect, the intersection volume V intersection =0, the user is prompted that the current gesture has a potential conflict, and the bounding box is highlighted to prompt the user to adjust the gesture position;

[0132] When gestures overlap and intersect, V intersection >0, the overlapping and intersecting areas are considered to be in conflict, and gesture processing in the conflicting area is suspended. The user is reminded by voice that there is a conflict in gestures and to make adjustments in a timely manner.

[0133] At this point, a gesture recognition method for flying screen interaction is completed.

[0134] See also Figure 2 , Figure 2 This is an overall architecture diagram of a gesture recognition system for flying screen interaction.

[0135] In this embodiment, a gesture recognition system for flying screen interaction includes the following modules:

[0136] The data acquisition module 21 is used to acquire video data of user gestures.

[0137] The background subtraction module 22 is used to process the video data using a background subtraction algorithm to obtain a foreground area in the video data.

[0138] The data extraction module 23 is used to extract the foreground area in the video data by using human posture estimation to obtain a set of key points of the hands of each human body.

[0139] The hand modeling module 24 is used to construct a hand skeleton model using forward kinematics based on a set of hand key points of each human body.

[0140] The simulation recognition module 25 is used to simulate the movement of the hand using the hand skeleton model and extract the gesture area and actual spatial position of each human body.

[0141] The comprehensive determination module 26 is used to determine whether the gestures of different users overlap or intersect in space based on the gesture area and actual spatial position of each human body through distance measurement combined with collision detection.

[0142] The feedback module 27 is used to trigger interactive feedback according to the judgment result.

[0143] The background subtraction module 22 mentioned above specifically includes:

[0144] The initialization unit is used to initialize the parameters of the Gaussian mixture model; the Gaussian mixture model parameters include: the number of Gaussian distributions, the learning rate and the foreground determination threshold.

[0145] The Gaussian mixture model construction unit is used to construct a Gaussian mixture model according to the initialized Gaussian mixture model parameters.

[0146] The multiple video frame extraction units are used to extract multiple video frames containing human portraits from the video data, each of the video frames containing multiple pixel points.

[0147] The background model construction unit is used to establish a background model corresponding to each pixel point using a Gaussian mixture model; the background model includes at least one Gaussian distribution.

[0148] The judgment unit is used to calculate the total matching degree corresponding to each pixel point based on each Gaussian distribution in the background model corresponding to each pixel point, and judge whether the current pixel point is a foreground area or a background area according to the total matching degree.

[0149] The background model updating unit is used to update the background model to adapt to the current scene changes if the current pixel point is considered to be the foreground area; if the current pixel point is considered to be the background, the background model is not updated and the current background model is still used.

[0150] The foreground mask image generating unit is used to mark the pixels in the foreground area of ​​each video frame as 1 and mark the pixels in the background area of ​​each video frame as 0, so as to generate a foreground mask image of each video frame.

[0151] The foreground mask image merging unit is used to merge the foreground mask images in each video frame, track and extract the foreground area in the video data.

[0152] The data extraction module 23 mentioned above specifically includes:

[0153] The limb key point recognition unit is used to identify the foreground area in the video data using a human posture estimation model to obtain multiple limb key points related to the human body in the video data; the multiple limb key points include the head, torso, limbs and hands.

[0154] The key point information determination unit is used to output key point information related to the hand according to the fixed index of the hand corresponding to each human body in the key point sequence; the key point information related to the hand includes the two-dimensional coordinates of the wrist and finger joint positions.

[0155] The hand key point set generating unit is used to generate a set of hand key points corresponding to each human body according to the key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

[0156] At this point, a gesture recognition system for flying screen interaction is completed.

[0157] The above description is merely a specific embodiment of the present invention, but the scope of protection of the embodiments of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention. Therefore, the scope of protection of the embodiments of the present invention should be based on the scope of protection of the claims.

Claims

1. A gesture recognition method for flying screen interaction, characterized in that: These include: Get video data of user gestures; The video data is processed using a background subtraction algorithm to obtain the foreground area in the video data; Human body posture estimation is used to extract the foreground area in the video data to obtain the set of key points of each human hand; Based on the set of key points of each human hand, the hand skeleton model is constructed using forward kinematics; Use the hand skeleton model to simulate hand movements and extract the gesture area and actual spatial position of each body; Based on the gesture area and actual spatial position of each body, distance measurement combined with collision detection is used to determine whether the gestures of different users overlap or intersect in space. Specifically, the following steps are performed: Calculate the center point distance between each human gesture based on the gesture area and actual spatial position of each human gesture; Determine whether the center point distance is less than the overlap threshold; if the center point distance is less than the overlap threshold, it is determined that the two gesture areas have preliminary overlap, otherwise there is no preliminary overlap; When two gesture areas have a preliminary overlap, the axis-aligned bounding box collision detection algorithm is used to determine whether the two bounding boxes have an intersection based on the spatial overlap condition. If there is an intersection, the two gesture areas are considered to have a spatial overlap; if there is no intersection, the two gesture areas are considered to have no spatial overlap. When two gesture areas overlap in space, the intersection volume of the two gesture areas is calculated to determine whether the intersection volume is greater than 0. If the intersection volume is greater than 0, it is determined that the two gesture areas have spatially intersected, otherwise it is considered that there is no spatial intersection. Trigger interactive feedback based on the judgment results.

2. A gesture recognition method for flying screen interaction according to claim 1, characterized in that: The background subtraction algorithm is used to process the video data to obtain the foreground area in the video data, including: Initialize the Gaussian mixture model parameters; the Gaussian mixture model parameters include: the number of Gaussian distributions, the learning rate, and the foreground determination threshold; Construct a Gaussian mixture model according to the initialized Gaussian mixture model parameters; Extracting multiple video frames containing human portraits from the video data, each of the video frames containing multiple pixels; A background model corresponding to each pixel is established using a Gaussian mixture model; the background model includes at least one Gaussian distribution; Calculate the total matching degree of each pixel based on the Gaussian distribution in the background model corresponding to each pixel, and determine whether the current pixel is in the foreground area or the background area based on the total matching degree; If the current pixel is considered to be the foreground area, the background model is updated to adapt to the current scene changes; if the current pixel is considered to be the background, the background model is not updated and the current background model is still used; Mark the pixels in the foreground area of ​​each video frame as 1, and mark the pixels in the background area of ​​each video frame as 0, to generate a foreground mask image of each video frame; The foreground mask images in each video frame are merged to track and extract the foreground area in the video data.

3. The gesture recognition method for flying screen interaction according to claim 1, characterized in that: Human pose estimation is used to extract the foreground area in the video data to obtain a set of key points of each human hand, including: A human posture estimation model is used to identify the foreground area in the video data and obtain multiple limb key points related to the human body in the video data; the multiple limb key points include the head, torso, limbs and hands; According to the fixed index of each hand corresponding to the human body in the key point sequence, the key point information related to the hand is output; the key point information related to the hand includes the two-dimensional coordinates of the wrist and finger joint positions; A set of key points of the hand corresponding to each human body is generated according to key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

4. The gesture recognition method for flying screen interaction according to claim 1, characterized in that: Based on the set of key points of each human hand, forward kinematics is used to construct a hand skeleton model, which includes: Define the connection relationship between joint nodes and joints based on the key point set of each human hand, and calculate the spatial transformation relationship between each joint; According to the spatial transformation relationship between each joint, a hand skeleton model is constructed, which includes the three-dimensional positions and relative motion relationships of all joints.

5. The gesture recognition method for flying screen interaction according to claim 1, characterized in that: The hand skeleton model is used to simulate hand movements and extract the gesture area and actual spatial position of each human body, including: A 3D bounding box is constructed based on all key 3D positions within the hand skeleton model. Within the 3D bounding box, the gesture area is calculated based on the relative motion relationship within the hand skeleton model. Based on the continuous frames in the video data, the changes of the hand skeleton model corresponding to each human body are tracked, and the motion trajectory of each joint corresponding to each human body in three-dimensional space is calculated; The actual position of each human hand in space is extracted according to the motion trajectory of each joint corresponding to each human body in three-dimensional space.

6. The gesture recognition method for flying screen interaction according to claim 1, characterized in that: Trigger interactive feedback based on the judgment results, including: When there is no spatial overlap between gestures, the center point distance is greater than the judgment threshold, or the bounding boxes have no intersection, and when there is no spatial overlap between gesture areas, each gesture area is considered independent, and the instructions of each gesture are processed independently without considering interference between users. Otherwise, the current state of the gesture is specifically judged and feedback is triggered.

7. A gesture recognition system for flying screen interaction, characterized in that: Includes the following modules: A data acquisition module, used to acquire video data of user gestures; A background subtraction module is used to process the video data using a background subtraction algorithm to obtain a foreground area in the video data; The data extraction module is used to extract the foreground area in the video data by using human posture estimation to obtain a set of key points of the hands of each human body; The hand modeling module is used to construct a hand skeleton model using forward kinematics based on the set of key points of each human hand; The simulation recognition module is used to simulate the hand movement using the hand skeleton model and extract the gesture area and actual spatial position of each human body; The comprehensive judgment module is used to determine whether the gestures of different users overlap or intersect based on the gesture area and actual spatial position of each body, through distance measurement combined with collision detection. Specifically, it includes: A center point calculation unit, configured to calculate the center point distance between the gestures of each human body according to the gesture area and the actual spatial position of each human body; A preliminary determination unit is used to determine whether the center point distance is less than the overlap threshold; if the center point distance is less than the overlap threshold, it is determined that the two gesture areas have preliminary overlap, otherwise there is no preliminary overlap; A collision detection unit is configured to, when two gesture areas have a preliminary overlap, use an axis-aligned bounding box collision detection algorithm to determine whether the two bounding boxes have an intersection based on the spatial overlap condition; if there is an intersection, the two gesture areas are determined to have a spatial overlap; if there is no intersection, the two gesture areas are determined to have no spatial overlap; The intersection confirmation unit is used to calculate the intersection volume of the two gesture areas when the two gesture areas overlap, and determine whether the intersection volume is greater than 0; if the intersection volume is greater than 0, it is determined that the two gesture areas have spatially overlapped, otherwise it is considered that there is no spatial overlap; The feedback module is used to trigger interactive feedback based on the judgment results.

8. The gesture recognition system for flying screen interaction according to claim 7, characterized in that: The background subtraction module specifically includes: An initialization unit, configured to initialize Gaussian mixture model parameters, including the number of Gaussian distributions, a learning rate, and a foreground determination threshold; A Gaussian mixture model construction unit, used to construct a Gaussian mixture model according to the initialized Gaussian mixture model parameters; Multiple video frame extraction units, used to extract multiple video frames containing human portraits from the video data, each of the video frames containing multiple pixels; A background model building unit, configured to build a background model corresponding to each pixel using a Gaussian mixture model; the background model includes at least one Gaussian distribution; a judgment unit, configured to calculate a total matching degree corresponding to each pixel point based on each Gaussian distribution in the background model corresponding to each pixel point, and to judge whether the current pixel point is in the foreground area or the background area according to the total matching degree; A background model updating unit is used to update the background model to adapt to the current scene change if the current pixel point is considered to be the foreground area; if the current pixel point is considered to be the background, the background model is not updated and the current background model is still used; a foreground mask image generating unit, configured to mark pixels in the foreground area of ​​each video frame as 1 and pixels in the background area of ​​each video frame as 0, thereby generating a foreground mask image of each video frame; The foreground mask image merging unit is used to merge the foreground mask images in each video frame, track and extract the foreground area in the video data.

9. The gesture recognition system for flying screen interaction according to claim 7, characterized in that: The data extraction module specifically includes: a limb key point recognition unit, configured to recognize a foreground region in the video data using a human posture estimation model, and obtain a plurality of limb key points related to the human body in the video data; the plurality of limb key points include the head, torso, limbs, and hands; A key point information determination unit is configured to output key point information related to the hand based on a fixed index of the hand corresponding to each human body in the key point sequence; the key point information related to the hand includes two-dimensional coordinates of the wrist and finger joint positions; The hand key point set generating unit is used to generate a set of hand key points corresponding to each human body according to the key point information related to the hand corresponding to each human body, and the set includes two-dimensional coordinate data of each key point of the hand.

Images