Method and system for intelligently capturing air gesture actions and processing interactive data

Abstract

The invention relates to the technical field of gesture recognition, in particular to an air gesture action intelligent capture and interactive data processing method and system.According to the air gesture action intelligent capture and interactive data processing method and system, significant jumps are marked through timestamps, time sequence deconstruction of action streams is achieved, nonlinear alignment processing of time structures between paragraphs is introduced, and the time sequence deconstruction efficiency is improved. The fragment consistency recognition capability under the conditions of unequal action durations and different rhythms is remarkably improved, spatial topology among three points is modeled by adopting a graph structure, a graph convolutional network is introduced to process dynamic changes of connection relations among different nodes, the continuity and stability of structure judgment can be kept under a shielding or structure deformation scene, and the recognition efficiency is improved. Through a triple constraint mechanism of direction vector included angle amplitude, length ratio and slope fluctuation, the action is subjected to modeling support in three aspects of time dimension, space structure and direction continuity at the same time, and the dynamic stability of gesture recognition and the resolution precision of paragraph judgment are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition technology, and in particular to a method and system for intelligently capturing and processing interactive data of gestures in the air. Background Art

[0002] The field of gesture recognition technology uses non-contact methods to identify gestures and body movements and convert them into control instructions that can be used for human-computer interaction. It is used in smart terminals, wearable devices, vehicle systems and interactive display platforms. Gesture recognition technology integrates multiple algorithms of computer vision, image processing, pattern recognition and machine learning. The goal is to achieve accurate understanding and instant response to human movements, and improve the naturalness and sensitivity of device interaction.

[0003] The method for intelligent capture and interactive data processing of air gestures aims to address the limitations of existing contact-based or traditional visual interaction methods, such as contact delay, high recognition error, and narrow coverage of interactive commands. By achieving stable capture and data processing of air gestures, the response accuracy and ease of use of devices in complex environments are improved. The goal is to establish a reliable and highly responsive input mechanism that allows users to complete command input, interface switching or parameter control without touching the device, thereby enhancing the device's intelligent interactive experience and operational efficiency.

[0004] Existing technologies lack deep modeling of temporal dynamic structures when processing continuous gestures, and the boundaries of actions are difficult to effectively define from time series, resulting in ambiguous recognition start and end points. In unlabeled environments, existing methods for segmenting continuous trajectories often rely on fixed time windows or simple threshold judgments, resulting in the inability to accurately extract short actions. Continuous actions often suffer from segmentation dislocation due to uneven frame numbers, and cannot accurately reflect the continuity of the actual action topology. Especially in interactive scenarios, when gestures are obscured or the direction of the action changes dramatically, unstable structural modeling can lead to misjudgment of the entire action. The lack of joint judgment of the change in action slope and spatial direction also leads to insufficient differentiation of the system in similar action segments, which can easily lead to similar misidentification. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a method and system for intelligently capturing and processing interactive data of gestures in the air.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligently capturing and processing interactive data of air gestures, comprising the following steps:

[0007] Step 1: Based on the position changes of the wrist, palm, and fingertips in consecutive frame images, the frame coordinates are extracted and the angle increment and distance difference between adjacent frames are calculated. The timestamps when the distance difference is greater than the preset range are recorded to generate a sequence of action direction offset paragraph markers.

[0008] Step 2: Based on the action direction offset paragraph mark sequence, the trajectory data is extracted and segmented according to the frame number range of the paragraph, and the time difference of the trajectory paragraphs is combined with the dynamic time warping and the sum of the distances of the continuous segments of the trajectory is calculated to establish a list of determinate action trajectory paths;

[0009] Step 3: Based on the list of identifiable motion trajectory paths, extract the set of directional vectors within the segment and calculate the total change in angle between the vectors and the change density in chronological order. Filter out segments where the total density is lower than the mutation threshold valley value, and output a high-change directional jump segment index group.

[0010] Step 4: Based on the high-variability direction jump segment index group, the coordinate points of the wrist, palm, and fingertips are extracted, the distance between the two points is calculated and connected as edges to construct an undirected graph structure, a graph convolutional network is introduced to model the node relationship, the total connectivity of the graph structure is calculated, and a set of action topology aggregation structures is formed;

[0011] Step 5: Based on the action topology aggregation structure set, extract the set of line segment direction vectors within the structure, calculate the angle change amplitude between adjacent directions and the ratio of adjacent length segments, calculate the mean slope fluctuation of similar segments in consecutive directions, and output a set of identifiable stable action structures.

[0012] As a further solution of the present invention, the specific steps of generating the action direction offset paragraph mark sequence are:

[0013] Based on the position changes of the wrist, palm, and fingertips in continuous frame images, the bounding box of the corresponding area in the image pixel array is cropped, and the two-dimensional coordinate values of the three points in the frame are extracted using coordinate mapping rules. The coordinate values are sorted in order and a sequence index is established in a numbered manner to generate a three-point coordinate frame sequence;

[0014] Based on the three-point coordinate frame sequence, a displacement vector of the same point in adjacent frames is constructed, and the angle change value of the line connecting the wrist and the palm between frames is calculated using the vector angle formula. The Euclidean distance difference between any two of the three points is calculated to obtain a list of frame-by-frame angle changes and establish a distance difference matrix to generate a sequence of direction and distance changes between frames;

[0015] Based on the inter-frame direction and distance change sequence, the frame angle change value and the distance difference are compared within the defined upper and lower threshold ranges, the frame numbers greater than the upper threshold are extracted, the mapping of the frame number set in the original timeline sequence is sorted out, and the action direction offset paragraph mark sequence is generated.

[0016] As a further solution of the present invention, the specific steps of establishing the list of decidable motion trajectory paths are:

[0017] Based on the motion direction offset paragraph mark sequence, extract the paragraph start frame number to generate a frame number index list, use the index to locate the start and end positions in the stored frame data, clip the three-point trajectory coordinates segment by segment, combine the coordinates in the order of the numbers and combine the three-point trajectories of the wrist, palm, and fingertip into independent coordinate sequences in the order of the frame numbers, establish an index correspondence between the trajectory point position and the frame number, and generate a paragraph trajectory data set;

[0018] Based on the paragraph trajectory data set, the three-point coordinate sequence in the paragraph is divided into equal intervals, point-pair connecting segments are constructed, the segment lengths are calculated and a segment length sequence list is established, a traversal mechanism is used to perform an item-by-item accumulation operation on the length sequence list, the cumulative path length of the paragraph trajectory is calculated and stored according to the original segment number, and dynamic time warping is then used to dynamically align the paragraph length sequence with the historical trajectory reference segment and calculate the time matching difference to generate a paragraph distance total sequence;

[0019] Based on the total distance sequence of the segment, the start and end frame numbers of the spliced segment are combined with the corresponding three-point coordinate trajectories, and a key-value structure is used to establish a mapping between the segment number and the trajectory segment. When establishing the mapping between the segment number and the trajectory segment, the corresponding trajectory cumulative distance data is written into the mapping structure as an additional attribute, completing the unified organizational structure of the segment number, continuous trajectory and cumulative distance, and generating a list of determinate action trajectory paths.

[0020] As a further solution of the present invention, the specific steps of outputting the high-change direction jump segment index group are:

[0021] Based on the list of decidable motion trajectory paths, coordinate points arranged in time order in the segment trajectory are extracted, connecting vectors are constructed by using two consecutive coordinate points, and each connecting vector is combined into a vector sequence in time order, and each connecting vector is combined into a vector sequence in time order, and stored in a list structure indexed in frame order to form a set of intra-segment direction vectors;

[0022] Based on the set of directional vectors within the segment, the angles between adjacent vectors are calculated in sequence according to the vector arrangement order, the directional angles formed by two adjacent vectors are processed using trigonometric functions and the angle values are accumulated, and the sum of the angles is divided by the number of frames within the corresponding segment to obtain the angle change per unit frame, thereby generating an angle change density sequence;

[0023] Based on the angle change density sequence, the paragraph density value is compared with the set jump threshold value, the jump threshold corresponding to the value in the angle change density sequence is compared item by item, the segment number index greater than and equal to the jump threshold is retained, the segment numbers that meet the conditions are summarized in sequence to form a jump segment index sequence, an ordered number output structure is constructed, and a high-change direction jump segment index group is generated.

[0024] As a further solution of the present invention, the specific steps of forming the action topology aggregation structure set are:

[0025] Based on the high-variability direction jump segment index group, the frame index corresponding to the jump segment is located and frame images are sequentially extracted in the order of the numbers. The hand area is located according to the frame image. After the boundary range of the hand in the image is determined, pixel coordinate data is extracted from the corresponding area to form a coordinate matrix. The coordinates of the wrist, palm, and fingertips are extracted and numbered and classified into a structure array respectively to generate a key point coordinate segment set.

[0026] Based on the key point coordinate segment set, the coordinates of the wrist, palm, and fingertips are combined into a group of two points to form a coordinate pair, the coordinate difference of each dimension of the coordinate pair is calculated and squared and then summed, and then a square root operation is performed to obtain the corresponding Euclidean distance value, a straight line segment is constructed between the two points using the coordinate index and the connection structure is recorded in frame order, the line segment set structure and the node binding index in the jump segment are obtained, and a multi-segment line segment construction atlas is generated;

[0027] An atlas is constructed based on the multiple line segments, the number of line segments connected to the coordinate points in the paragraph graph structure is counted and a connection number index table is established according to the node position, the number of connecting edges of each node in the graph structure is counted as the connectivity of the node, and the connectivity of each node is summed to obtain the total connectivity value of the graph structure. Combined with the graph convolutional network, the connectivity of each node is used as the input feature value to model the association relationship between the nodes in the graph structure, complete the mapping of structure number and connectivity set organization, and generate an action topology aggregation structure set.

[0028] As a further solution of the present invention, the specific steps of outputting the identifiable stable action structure set are:

[0029] Based on the action topology aggregation structure set, extract the line segment node index pairs in the graph structure and generate directional coordinate differences according to the start and end coordinate points. Use the directional coordinate differences to convert them into a vector structure in sequence according to the frame number. All vector sets are assigned to corresponding index positions according to the segment number and a segment mapping structure is established to generate a directional vector set within the structure.

[0030] Based on the direction vector set within the structure, the angles between consecutive vectors in the vector set are calculated and a sequence of angle values is generated. Two consecutive vectors are paired in the form of a ratio of the angle change amplitude to the length difference of the corresponding line segments and the corresponding serial number positions are recorded. The angle change and length ratio sequences are sorted by segment number to generate a sequence of direction change and length ratio.

[0031] Based on the direction change and length ratio sequence, the slope difference between adjacent elements in the sequence is calculated, the mean of continuous slope changes is counted in the fixed window traversal interval, the fluctuation stable segment is extracted by mean sorting, and a stable segment number structure table is established to generate a recognizable stable action structure set.

[0032] A system for intelligently capturing and processing data of gestures in the air, which is used to execute the above-mentioned method for intelligently capturing and processing data of gestures in the air, comprises:

[0033] The motion monitoring module extracts the coordinates of the wrist, palm, and fingertips frame by frame based on the changes in the coordinates of the three points on the time axis in consecutive frame images. It calculates the angle increment and Euclidean distance difference between the lines formed by the three points in adjacent frames. It records the frame numbers with inter-frame distance differences greater than a preset range and maps them to the corresponding time axis positions, generating a sequence of motion direction offset paragraph markers.

[0034] Trajectory segmentation module: Based on the action direction offset paragraph mark sequence, the three-point coordinate data in the corresponding frame image is extracted according to the paragraph frame number interval, and a paragraph trajectory coordinate set is sequentially constructed. After the trajectory is divided into segments with equal intervals, the distance between the point pairs is calculated to form a distance sequence. The distance values of each trajectory segment are accumulated. In combination with the dynamic time warping method, the trajectory segments are time-aligned with the reference trajectory and the total distance data is generated to establish a list of determinate action trajectory paths;

[0035] Density screening module: Based on the list of identifiable motion trajectory paths, it extracts the coordinate points of each trajectory segment arranged in chronological order and constructs a set of direction vectors. It calculates the total amount of angle change between adjacent vectors and calculates the angle change density per frame based on the number of frames. It removes segments with a total density below the set jump threshold, retains the high-density segment numbers, and outputs a high-change direction jump segment index group.

[0036] Structure construction module: Based on the high-variability direction jump segment index group, the coordinate points of the wrist, palm, and fingertips in the corresponding segment are extracted, the Euclidean distance is calculated for each pair of points, and edges are constructed based on the connection relationship. An undirected graph structure is established based on the coordinate points, with the coordinate points set as nodes and the edges representing the connection relationship between the two points. The number of edges of each node is counted and the total connectivity is calculated. A graph convolutional network is introduced, and the connection relationship between nodes is used as input to model the graph structure, generating a set of action topology aggregation structures;

[0037] Action recognition module: Based on the action topology aggregation structure set, the direction vector set corresponding to each line segment in the graph structure is extracted, the angle change amplitude between adjacent direction vectors and the ratio of the corresponding line segment length are calculated, and a direction change and length ratio sequence is constructed. The continuous slope fluctuation mean of the sequence is calculated within a fixed sliding window, and the stable fluctuation segment is identified by mean sorting. The stable segment number is extracted and a structure index is established, and a set of identifiable stable action structures is output.

[0038] Compared with the prior art, the advantages and positive effects of the present invention are:

[0039] 1. This invention achieves temporal deconstruction of the action flow by marking significant jumps with timestamps. It combines trajectory data with a dynamic time warping mechanism and introduces nonlinear alignment of the temporal structure between segments. This significantly improves the ability to identify segment consistency even when the action durations and rhythms are different.

[0040] 2. This invention uses a graph structure to model the spatial topology between three points and introduces a graph convolutional network to handle the dynamic changes in the connection relationships between different nodes. This can maintain the consistency and stability of structural judgment in scenarios with occlusion or structural deformation, thereby improving the ability to accurately identify effective action areas in gesture streams.

[0041] 3. In the present invention, a triple constraint mechanism of directional vector angle amplitude, length ratio, and slope fluctuation is used to simultaneously support modeling of actions at the three levels of time dimension, spatial structure, and directional continuity, thereby enhancing the dynamic stability of gesture recognition and the resolution accuracy of paragraph determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0043] Figure 2 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0046] Example 1

[0047] See also Figure 1 The present invention provides a technical solution: a method for intelligently capturing and processing interactive data of air gestures, comprising the following steps:

[0048] Step 1: Based on the position changes of the wrist, palm, and fingertips in consecutive frame images, the frame coordinates are extracted and the angle increment and distance difference between adjacent frames are calculated. The timestamps when the distance difference is greater than the preset range are recorded to generate a sequence of action direction offset paragraph markers.

[0049] Step 2: Based on the action direction offset paragraph marker sequence, the trajectory data is extracted and segmented according to the paragraph frame number range. The time difference of the trajectory segments is combined with dynamic time warping and the sum of the distances of consecutive segments of the trajectory is calculated to establish a list of determinate action trajectory paths;

[0050] Step 3: Based on the list of identifiable motion trajectory paths, extract the set of directional vectors within the segment and calculate the total change and change density of the angles between the vectors in chronological order. Segments with a total density below the mutation threshold are filtered out, and the index group of high-change directional jump segments is output.

[0051] Step 4: Based on the high-variability directional jump segment index group, the coordinate points of the wrist, palm, and fingertips are extracted. The distance between the two points is calculated and connected as edges to construct an undirected graph structure. A graph convolutional network is introduced to model the node relationships. The total connectivity of the graph structure is calculated to form a set of action topology aggregation structures.

[0052] Step 5: Based on the action topology aggregation structure set, extract the set of line segment direction vectors within the structure, calculate the angle change amplitude between adjacent directions and the ratio of adjacent length segments, calculate the mean slope fluctuation of similar segments in consecutive directions, and output a set of identifiable stable action structures.

[0053] The specific steps for generating an action direction offset paragraph mark sequence are:

[0054] Based on the position changes of the wrist, palm, and fingertips in continuous frame images, the bounding box of the corresponding area in the image pixel array is cropped, and the two-dimensional coordinate values of the three points in the frame are extracted using coordinate mapping rules. The coordinate values are sorted in order and a sequence index is established in a numbered manner to generate a three-point coordinate frame sequence;

[0055] Based on a three-point coordinate frame sequence, the displacement vector of the same point in adjacent frames is constructed. The angle change between the wrist and the palm of the hand between frames is calculated using the vector angle formula. The Euclidean distance difference between any two of the three points is calculated. This generates a list of frame-by-frame angle changes and a distance difference matrix, generating a sequence of direction and distance changes between frames.

[0056] Based on the inter-frame direction and distance change sequence, the frame angle change value and the distance difference are compared within the defined upper and lower threshold ranges, the frame numbers greater than the upper threshold are extracted, the mapping of the frame number set in the original timeline sequence is sorted out, and the action direction offset paragraph mark sequence is generated;

[0057] Based on the position changes of the wrist, palm, and fingertips in continuous frame images, the image bounding box extraction method is adopted to scan the three-point area in the image pixel array. By setting the contour extreme value range of each point, a pixel index set containing the upper, lower, left, and right boundaries is constructed. The regional pixel cropping is performed and the corresponding frame number is retained. Then, using the two-dimensional coordinate mapping rule, the image origin is set to the upper left corner, and the horizontal and vertical pixel positions of the key points are mapped to the x- and y-axis values respectively. A coordinate group sequence arranged in chronological order is constructed. Different frames are distinguished by number identifiers and a continuous index table is established to generate a three-point coordinate frame sequence. This improves the continuity of target area extraction in the image frame and the temporal correlation of the frame index.

[0058] Based on a three-point coordinate frame sequence, the displacement vector structure between the three points in adjacent frames is constructed. The inter-frame vector difference method is used to select the same key point in two adjacent frames on the time axis. The differences in the horizontal and vertical coordinates are calculated and combined into a direction vector. The direction angle measurement method is used to compare the degree of directional offset of the two frame vectors from the wrist to the palm. The direction offset reference range is set to 0 to 180 degrees to form an angle change array. Then, a two-point Euclidean distance evaluation mechanism is used to define the point pair combination order. The distance increment calculation is performed on each pair of points between the three points and the matrix is organized into a frame sequence matrix. The inter-frame direction and distance change sequence is generated. This enhances the modeling accuracy of continuous action trends in the time dimension and the expressiveness of spatial distance changes.

[0059] Based on the direction and distance change sequence between frames, a dual-threshold paragraph mutation judgment mechanism is adopted. The angle change threshold is set to 20 degrees and the distance change threshold is set to 15 unit pixels. The angle and distance difference values of each frame are compared in sequence, and the frame numbers with values exceeding the set threshold are retained. The numbers are deduplicated and arranged in chronological order. The original index data of the previous image frame is called to perform number mapping to obtain a list of continuous paragraph markers, and a sequence of action direction offset paragraph markers is generated. This enhances the ability to capture the temporal sequence of sudden action trends in continuous frames and the accuracy of identifying the start and end breakpoints of the target segment.

[0060] The specific steps to establish a list of decidable motion trajectory paths are:

[0061] Based on the motion direction offset paragraph mark sequence, the paragraph start frame number is extracted to generate a frame number index list. The index is used to locate the start and end positions in the stored frame data. The three-point trajectory coordinates are clipped segment by segment, and the coordinates are combined and aligned in the order of the numbers. The three-point trajectories of the wrist, palm, and fingertips are respectively combined into independent coordinate sequences in the order of the frame numbers. The index correspondence between the trajectory point position and the frame number is established to generate a paragraph trajectory data set.

[0062] Based on the paragraph trajectory data set, the three-point coordinate sequence in the paragraph is divided into equal intervals, and line segments connecting point pairs are constructed. The line segment lengths are calculated and a segment length sequence list is established. A traversal mechanism is used to perform an item-by-item accumulation operation on the length sequence list. The cumulative path length of the paragraph trajectory is calculated and stored according to the original segment number. Dynamic time warping is then used to dynamically align the paragraph length sequence with the historical trajectory reference segment and calculate the time matching difference to generate a paragraph distance total sequence.

[0063] Based on the total distance sequence of the segment, the start and end frame numbers of the spliced segment are combined with the corresponding three-point coordinate trajectory. A key-value structure is used to establish a mapping between the segment number and the trajectory segment. When establishing the mapping between the segment number and the trajectory segment, the corresponding trajectory cumulative distance data is written into the mapping structure as an additional attribute. This completes the unified organization structure of the segment number, continuous trajectory and cumulative distance, and generates a list of determinate action trajectory paths.

[0064] Based on the motion direction offset paragraph marker sequence, the paragraph start frame number and end frame number are set as index boundaries, and a number interval list is constructed as index input. The corresponding image data in the image frame buffer is called in frame number sequence. The coordinates of the wrist, palm, and fingertips are extracted using the structure coordinate index field in the image buffer. The coordinate data are numbered in sequence using a sequential numbering rule. The data is combined by segment using a structure array format and a number corresponding dictionary structure is established to generate a paragraph trajectory data set. This improves the frame positioning accuracy of trajectory segment extraction and the efficiency of key point structure construction.

[0065] Based on the segment trajectory data set, a line segmentation and length aggregation method is adopted. The point pair connection rule is set to construct connecting segments in the order of adjacent frames among the three points in each segment. The segmentation length is set to a fixed frame interval. The point pair connections are traversed and the segment lengths are counted according to the coordinate difference to form a corresponding length sequence list. Then, a linear accumulation strategy is adopted to accumulate and store the segment lengths in the order of numbering. A dynamic time warping algorithm is used. The reference trajectory segment is set as the standard template. The length sequence of each segment is compared with the template sequence item by item through a sliding window method and the matching offset value is recorded. The difference of all step lengths in the matching path is counted as the time alignment difference to generate a total segment distance sequence. This enhances the consistency expression and comparison ability of trajectory segment length under the temporal structure.

[0066] Based on the total distance sequence of each segment, a structure aggregation and mapping construction method is adopted to associate the start and end frame numbers of each segment with the corresponding three-point trajectory coordinates of the wrist, palm, and fingertips. Combined structural fragments are generated and a mapping table between segment numbers and trajectory structures is constructed. The key-value storage mechanism is used to write each segment trajectory structure according to the number, and the corresponding cumulative distance value of the trajectory is attached. This forms a ternary structure set consisting of the trajectory number, the three-point coordinate structure, and the total path distance. A list of identifiable action trajectory paths is generated, which improves the number mapping consistency and positioning management efficiency of multiple trajectory segments in the recognition process.

[0067] Dynamic Time Warping algorithm, according to the formula:

[0068] in: For the paragraph The length of the line segment between the frame trajectory points, For the template The standard length between frame trajectory points, For the paragraph The difference value of the frame trajectory length, that is , For the template The difference value of the frame trajectory length, that is , The main distance item weight is set to 0.7. is the differential item response weight, set to 0.3, For location The cumulative time alignment difference at , To be the first Frame trace and When the frame template is aligned, the cumulative distance on the path extending from the previous trajectory frame, To be the first Frame trace and When aligning frame templates, the cumulative distance on the path extending from the previous template frame, To be the first Frame trace and When aligning the frame template, the cumulative distance on the path from the previous trajectory frame and the template frame moving forward simultaneously;

[0069] Execution process: First, the coordinate data of the wrist, palm and fingertips in the continuous frame images are extracted from each segment, and the connecting line segments of adjacent coordinate points are constructed according to the frame sequence. The length of the line segments between each frame is calculated and the trajectory length sequence is generated according to the sequence number and recorded as , synchronously construct the standard template trajectory length sequence and record it as , then perform first-order difference operations on the two length sequences respectively to construct the difference sequence and , which indicates the changing trend of trajectory length in local time series. In the matching process, the index position of each frame is Calculate the principal distance terms separately and the differential response term , and according to the weight coefficient 、 Perform weighted superposition to form the current position matching cost, which is used as the corresponding position in the cumulative alignment value matrix The basic value of , select the smallest cumulative distance value from the left, bottom, and lower left directions of the matrix in turn, add it to the current matching cost, recursively construct the cumulative cost matrix of the entire alignment path, until the end position of the lower right corner of the matrix, and output the value of the end cell of the matrix , which is the sum of the minimum alignment step differences between the segment trajectory and the standard template in the complete frame sequence, generates a total segment distance sequence, and improves the matching accuracy of the trajectory segment to the standard template in the time dimension.

[0070] The specific steps for outputting the high change direction jump segment index group are:

[0071] Based on the path list of decidable motion trajectories, the coordinate points arranged in time order in the paragraph trajectory are extracted, and the connection vectors are constructed by using two consecutive coordinate points. The connection vectors are combined into a vector sequence in time order and stored in a list structure indexed in frame order to form a set of intra-segment direction vectors;

[0072] Based on the set of direction vectors within the segment, the angles between adjacent vectors are calculated in order of vector arrangement. The direction angles formed by two adjacent vectors are processed using trigonometric functions and the angle values are accumulated. The sum of the angles is divided by the number of frames within the corresponding segment to obtain the angle change per unit frame, generating an angle change density sequence.

[0073] Based on the angle change density sequence, the segment density value is compared with the set jump threshold value. The jump threshold corresponding to the value in the angle change density sequence is compared item by item. The segment number indexes greater than or equal to the jump threshold are retained. The segment numbers that meet the conditions are sequentially summarized to form a jump segment index sequence. An ordered number output structure is constructed to generate a jump segment index group with high change direction.

[0074] Based on a list of decidable motion trajectory paths, the method of generating adjacent frame coordinate vectors is adopted. The coordinate point sequence in each segment of the trajectory is slid, and two adjacent coordinate points are extracted as a group in the order of frame numbers. The horizontal and vertical coordinates of the starting and ending points are recorded to construct a two-dimensional direction vector structure. All vector segments are stored in a list format, and the vector start and end frame numbers are annotated for number index synchronization. This generates a set of intra-segment direction vectors, which improves the structured expression ability of motion direction under temporal continuity.

[0075] Based on the intra-segment directional vector set, a directional offset measurement method is used. Two continuous vector structures are extracted from the vector sequence in each segment in the order of number. The directional offset values between the starting and ending directions of the two segments are analyzed group by group. The directional offset value is obtained by comparing the horizontal and vertical offset amplitudes of each vector. The offset values are then accumulated according to the sequence of vectors in the segment, and the total amount of cumulative directional change is recorded. The total number of frames in the segment is set as the counting unit. The directional change density per frame is calculated and classified into the density result list under the corresponding segment number. This generates an angle change density sequence, which enhances the time-scale quantification ability of the directional intensity of the action segment.

[0076] Based on the angle change density sequence, a density threshold filtering method is adopted. The density screening standard is set as a fixed numerical threshold and applied to the segment number density result. The density value of each paragraph is compared to determine whether it is greater than or equal to the set density standard value. If the condition is met, the number is extracted and added to the screening number set. The set is sorted in the order of segment numbers and duplicate identifiers are removed. The segment number sequence that meets the requirements is output, and a high-change direction jump segment index group is generated, which improves the accurate screening ability of the jump action segmentation period.

[0077] The specific steps to form the action topology aggregation structure set are:

[0078] Based on the high-variability direction jump segment index group, the frame index corresponding to the jump segment is located and frame images are extracted in order of the numbers. The hand area is located according to the frame image. After determining the boundary range of the hand in the image, pixel coordinate data is extracted from the corresponding area to form a coordinate matrix. The coordinates of the wrist, palm, and fingertips are extracted and numbered and classified into a structure array to generate a key point coordinate segment set.

[0079] Based on the key point coordinate segment set, the coordinates of the wrist, palm, and fingertips are combined into coordinate pairs in pairs. The coordinate differences of each dimension of the coordinate pairs are calculated, squared, and summed. Then, a square root operation is performed to obtain the corresponding Euclidean distance value. The coordinate index is used to construct a straight line segment between the two points and the connection structure is recorded in frame order. The line segment set structure and node binding index in the jump segment are obtained to generate a multi-segment line segment construction atlas.

[0080] Based on multiple line segments, an atlas is constructed. The number of line segments connected to the coordinate points in the paragraph graph structure is counted and a connection number index table is established according to the node position. The number of connected edges of each node in the graph structure is counted as the node connectivity, and the connectivity of each node is summed to obtain the total connectivity value of the graph structure. In combination with the graph convolutional network, the connectivity of each node is used as the input feature value to model the association relationship between nodes in the graph structure. The structure number and connectivity set organization mapping are completed to generate the action topology aggregation structure set;

[0081] Based on the high-variability direction jump segment index group, an image frame key point extraction method is adopted. The image frame data in the numbered set is called sequentially, and the numbering order is used as the image frame extraction condition. After loading the frame image, the frame image is divided into multiple area blocks using a sliding window recognition method. The window size is set to 128 pixels. The hand area is identified using the area center grayscale and edge gradient threshold. The wrist, palm, and fingertip points are extracted from each area in sequence. The relative position encoding is used to mark the three-point index number. The extracted three-point data is encapsulated into a structure array form to generate a key point coordinate segment set, which enhances the stability of the three-point coordinate structure extraction in multi-segment images.

[0082] Based on the key point coordinate paragraph set, a graph structure line segment generation method is adopted. Two points are selected from the three-point combination in each segment to form a point pair. A connection sequence table is constructed according to the time series. The pixel displacement difference of the coordinates in the horizontal and vertical directions is calculated and stored as the basic attribute value of the line segment in a temporary line segment structure. Then, a line segment set is formed according to the segment number. The start and end frame numbers and number indexes corresponding to the line segment are recorded using a sequential mapping. The connection relationship between points and line segments is bound and saved in a dictionary structure. The multi-segment line segment construction atlas is generated, which enhances the connection integrity of the point-line relationship in cross-frame paragraphs.

[0083] Based on multi-segment line segments, an atlas is constructed. The graph connection metric analysis method is adopted to perform fixed numbering processing on the nodes in each segment graph structure. After scanning the node index table, the number of times the node is referenced by the line segment in the corresponding segment is counted. The number of node connections is used to form a connection number matrix table, and a one-to-one mapping structure between the number and the connection value is established. The graph convolutional network structure encoding method is introduced, and a three-layer network model is set. The number of connections between adjacent nodes is used as the input feature for weighted aggregation processing. The output node dimension of the convolutional layer is set to 64, and the activation function is set to ReLU. The node output encoding value in the graph structure is used as the segment graph representation to obtain the structural mapping and generate the action topology aggregation structure set, which enhances the expression ability of the multi-segment graph model for encoding spatial connection characteristics.

[0084] Graph convolutional network, according to the formula:

[0085] in: is the sum of the adjacency matrix and the unit matrix of the paragraph graph structure, is the node degree matrix corresponding to the adjacency matrix, For the The node feature matrix of the layer is initially an input matrix consisting of the number of node connections. For graph convolution The trainable weight matrix of the layer, is the activation function, set to ReLU, 、 is the weighting coefficient, which controls the combined weight of the main graph convolution path and the auxiliary feature path. is the node position disturbance matrix calculated based on the key point displacement statistics, is the screening mask matrix generated by the node structural stability analysis model, For the The node feature matrix output by the layer, is the negative half power of the degree matrix, that is, the degree of the node Take the inverse square root and construct a diagonal matrix;

[0086] Execution process: First, take all nodes in the paragraph graph structure as units, perform node number fixing processing, scan the node index table, count the number of times each node is referenced by the line segment in the segment, and build the initial connection number matrix as the input feature matrix of the graph convolution network. Then, build the corresponding adjacency matrix for each segment and add the unit matrix to generate , combined with the node connectivity to construct a normalized degree matrix , perform three-layer graph convolution network propagation calculation, where the convolution dimension of each layer is set to 64, and the activation function is set to ReLU. In each layer of convolution, the main graph structure path and the auxiliary perturbation path are respectively feature aggregated according to the improved propagation formula. The main path relies on the node connection relationship to transfer weights, and the auxiliary path is generated by the perturbation matrix generated by the position difference between key points in the frames. and structural stability screening mask matrices Construction, main path weight coefficient Set to 0.75, the auxiliary path coefficient It is set to 0.25, extracts the encoding results of all nodes in the output layer and organizes them into an index mapping structure between segment numbers and feature vectors, generates an action topology aggregation structure set, and improves the encoding integrity and structural discrimination of multi-segment graphs in capturing topological changes in action space.

[0087] The specific steps to output a set of identifiable stable action structures are:

[0088] Based on the action topology aggregation structure set, the line segment node index pairs in the graph structure are extracted and the direction coordinate difference is generated according to the start and end coordinate points. The direction coordinate difference is converted into a vector structure in sequence according to the frame number. All vector sets are assigned to the corresponding index position according to the segment number and a segment mapping structure is established to generate the direction vector set within the structure.

[0089] Based on the direction vector set within the structure, the angle between consecutive vectors in the vector set is calculated and a sequence of angle values is generated. The angle change amplitude of two consecutive vectors is matched with the difference ratio of the corresponding line segment lengths and the corresponding serial number positions are recorded. The angle change and length ratio sequences are sorted by segment number to generate a sequence of direction change and length ratio.

[0090] Based on the direction change and length ratio sequence, the slope difference between adjacent elements in the sequence is calculated. The mean of continuous slope changes is calculated in the fixed window traversal interval. The mean is sorted to extract the fluctuation stable segment and establish a stable segment number structure table to generate a set of identifiable stable action structures.

[0091] Based on the action topology aggregation structure set, the coordinate difference conversion method is used to traverse the line segment node index pairs in the graph structure, extract the two-dimensional coordinate values of each group's starting and ending points one by one, and uniformly organize and process them according to the frame number sequence. The line segment node coordinate information in the structure is extracted in sequence according to the preset frame number sequence. A direction difference set is generated by constructing a coordinate point difference table. Subsequently, the coordinate differences of each direction are filled into the corresponding positions of their respective segment numbers, and a one-to-one corresponding segment index structure is constructed to generate a direction vector set within the structure, which improves the spatial sequence coherence of the subsequent angle change accuracy calculation.

[0092] Based on the set of directional vectors within the structure, a method of pairing the directional angle amplitude with the length difference ratio is adopted. Each segment of data is traversed in the order of the vector set, and the angular change amplitude formed by two consecutive directional vectors is extracted item by item. The length difference of the preceding and following segments is calculated in sequence. The angular amplitude change value and the corresponding length difference are paired and bound to the same serial number index. The angle change sequence and length ratio sequence are constructed based on the segment number as the classification basis. The mapping structure between the segment number and the sequence data is obtained, and the directional change and length ratio sequence are generated, which enhances the parsability of the geometric change trend at the paragraph level.

[0093] Based on the direction change and length ratio sequence, the slope change mean extraction method is adopted. The sliding window interval range is set to 5 frames and the step size is set to 1 frame. The angle change and length ratio combination sequence within each segment number is traversed, and the continuous value differences in the window are extracted in sequence. The mean of the slope difference within the window is calculated and the corresponding starting frame position number is recorded. Each segment number is matched with the average slope change value, and the segments are sorted from small to large according to the slope mean. The segment numbers corresponding to the top 30% of the sorting results are taken as stable segment representatives. This generates a set of identifiable stable action structures, which improves the accuracy of interval judgment in the structural stability identification process.

[0094] See also Figure 2 , a system for intelligent capture and interactive data processing of air gestures, the system includes:

[0095] The motion monitoring module extracts the coordinates of the wrist, palm, and fingertips frame by frame based on the changes in the coordinates of the three points on the time axis in consecutive frame images. It calculates the angle increment and Euclidean distance difference between the lines formed by the three points in adjacent frames. It records the frame numbers with inter-frame distance differences greater than a preset range and maps them to the corresponding time axis positions, generating a sequence of motion direction offset paragraph markers.

[0096] Trajectory Segmentation Module: Based on the action direction offset paragraph marker sequence, the three-point coordinate data in the corresponding frame image is extracted according to the paragraph frame number interval. The segment trajectory coordinate set is sequentially constructed. After the trajectory is divided into segments with equal intervals, the distance between the point pairs is calculated to form a distance sequence. The distance values of each segment are accumulated. In combination with the dynamic time warping method, the trajectory segments are time-aligned with the reference trajectory and the total distance data is generated to establish a list of determinate action trajectory paths.

[0097] Density screening module: Based on the list of identifiable motion trajectory paths, it extracts the coordinate points of each trajectory segment arranged in chronological order and constructs a set of direction vectors. It calculates the total angle change between adjacent vectors and uses the frame count to calculate the angle change density per frame. Segments with a total density below the set jump threshold are removed, and the high-density segment numbers are retained. The index group of high-change direction jump segments is output;

[0098] Structure construction module: Based on the high-variability direction jump segment index group, the coordinate points of the wrist, palm, and fingertips in the corresponding segment are extracted, the Euclidean distance is calculated for each pair of points, and edges are constructed based on the connection relationship. An undirected graph structure is established based on the coordinate points, with the coordinate points set as nodes and the edges representing the connection relationship between the two points. The number of edges of each node is counted and the total connectivity is calculated. A graph convolutional network is introduced, and the connection relationship between nodes is used as input to model the graph structure, generating a set of action topology aggregation structures;

[0099] Action recognition module: Based on the action topology aggregation structure set, the direction vector set corresponding to each line segment in the graph structure is extracted, the angle change amplitude between adjacent direction vectors and the ratio of the corresponding line segment length are calculated, and a direction change and length ratio sequence is constructed. The continuous slope fluctuation mean of the sequence is calculated within a fixed sliding window, and the stable fluctuation segment is identified by mean sorting. The stable segment number is extracted and a structure index is established, and the recognizable stable action structure set is output.

[0100] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for intelligently capturing and processing interactive data of air gestures, characterized in that: The following steps are involved: Step 1: Based on the position changes of the wrist, palm, and fingertips in consecutive frame images, the frame coordinates are extracted and the angle increment and distance difference between adjacent frames are calculated. The timestamps when the distance difference is greater than the preset range are recorded to generate a sequence of action direction offset paragraph markers. Step 2: Based on the action direction offset paragraph mark sequence, the trajectory data is extracted and segmented according to the frame number range of the paragraph, and the time difference of the trajectory paragraphs is combined with the dynamic time warping and the sum of the distances of the continuous segments of the trajectory is calculated to establish a list of determinate action trajectory paths; Step 3: Based on the list of identifiable motion trajectory paths, extract the set of directional vectors within the segment and calculate the total change in angle between the vectors and the change density in chronological order. Filter out segments where the total density is lower than the mutation threshold valley value, and output a high-change directional jump segment index group. Step 4: Based on the high-variability direction jump segment index group, the coordinate points of the wrist, palm, and fingertips are extracted, the distance between the two points is calculated and connected as edges to construct an undirected graph structure, a graph convolutional network is introduced to model the node relationship, the total connectivity of the graph structure is calculated, and a set of action topology aggregation structures is formed; Step 5: Based on the action topology aggregation structure set, extract the set of line segment direction vectors within the structure, calculate the angle change amplitude between adjacent directions and the ratio of adjacent length segments, calculate the mean slope fluctuation of similar segments in consecutive directions, and output a set of identifiable stable action structures.

2. The method for intelligently capturing and processing interactive data of air gestures according to claim 1, characterized in that: The specific steps of generating the action direction offset paragraph mark sequence are: Based on the position changes of the wrist, palm, and fingertips in continuous frame images, the bounding box of the corresponding area in the image pixel array is cropped, and the two-dimensional coordinate values of the three points in the frame are extracted using coordinate mapping rules. The coordinate values are sorted in order and a sequence index is established in a numbered manner to generate a three-point coordinate frame sequence; Based on the three-point coordinate frame sequence, a displacement vector of the same point in adjacent frames is constructed, and the angle change value of the line connecting the wrist and the palm between frames is calculated using the vector angle formula. The Euclidean distance difference between any two of the three points is calculated to obtain a list of frame-by-frame angle changes and establish a distance difference matrix to generate a sequence of direction and distance changes between frames; Based on the inter-frame direction and distance change sequence, the frame angle change value and the distance difference are compared within the defined upper and lower threshold ranges, the frame numbers greater than the upper threshold are extracted, the mapping of the frame number set in the original timeline sequence is sorted out, and the action direction offset paragraph mark sequence is generated.

3. The method for intelligently capturing and processing interactive data of air gestures according to claim 1, characterized in that: The specific steps of establishing a list of identifiable motion trajectory paths are: Based on the motion direction offset paragraph mark sequence, extract the paragraph start frame number to generate a frame number index list, use the index to locate the start and end positions in the stored frame data, clip the three-point trajectory coordinates segment by segment, combine the coordinates in the order of the numbers and combine the three-point trajectories of the wrist, palm, and fingertip into independent coordinate sequences in the order of the frame numbers, establish an index correspondence between the trajectory point position and the frame number, and generate a paragraph trajectory data set; Based on the paragraph trajectory data set, the three-point coordinate sequence in the paragraph is divided into equal intervals, point-pair connecting segments are constructed, the segment lengths are calculated and a segment length sequence list is established, a traversal mechanism is used to perform an item-by-item accumulation operation on the length sequence list, the cumulative path length of the paragraph trajectory is calculated and stored according to the original segment number, and dynamic time warping is then used to dynamically align the paragraph length sequence with the historical trajectory reference segment and calculate the time matching difference to generate a paragraph distance total sequence; Based on the total distance sequence of the segment, the start and end frame numbers of the spliced segment are combined with the corresponding three-point coordinate trajectories, and a key-value structure is used to establish a mapping between the segment number and the trajectory segment. When establishing the mapping between the segment number and the trajectory segment, the corresponding trajectory cumulative distance data is written into the mapping structure as an additional attribute, completing the unified organizational structure of the segment number, continuous trajectory and cumulative distance, and generating a list of determinate action trajectory paths.

4. The method for intelligently capturing and processing interactive data of air gestures according to claim 1, characterized in that: The specific steps of outputting the high change direction jump segment index group are: Based on the list of decidable motion trajectory paths, coordinate points arranged in time order in the segment trajectory are extracted, connecting vectors are constructed by using two consecutive coordinate points, and each connecting vector is combined into a vector sequence in time order, and each connecting vector is combined into a vector sequence in time order, and stored in a list structure indexed in frame order to form a set of intra-segment direction vectors; Based on the set of directional vectors within the segment, the angles between adjacent vectors are calculated in sequence according to the vector arrangement order, the directional angles formed by two adjacent vectors are processed using trigonometric functions and the angle values are accumulated, and the sum of the angles is divided by the number of frames within the corresponding segment to obtain the angle change per unit frame, thereby generating an angle change density sequence; Based on the angle change density sequence, the paragraph density value is compared with the set jump threshold value, the jump threshold corresponding to the value in the angle change density sequence is compared item by item, the segment number index greater than and equal to the jump threshold is retained, the segment numbers that meet the conditions are summarized in sequence to form a jump segment index sequence, an ordered number output structure is constructed, and a high-change direction jump segment index group is generated.

5. The method for intelligently capturing and processing interactive data of air gestures according to claim 1, characterized in that: The specific steps of forming the action topology aggregation structure set are: Based on the high-variability direction jump segment index group, the frame index corresponding to the jump segment is located and frame images are sequentially extracted in the order of the numbers. The hand area is located according to the frame image. After the boundary range of the hand in the image is determined, pixel coordinate data is extracted from the corresponding area to form a coordinate matrix. The coordinates of the wrist, palm, and fingertips are extracted and numbered and classified into a structure array respectively to generate a key point coordinate segment set. Based on the key point coordinate segment set, the coordinates of the wrist, palm, and fingertips are combined into a group of two points to form a coordinate pair, the coordinate difference of each dimension of the coordinate pair is calculated and squared and then summed, and then a square root operation is performed to obtain the corresponding Euclidean distance value, a straight line segment is constructed between the two points using the coordinate index and the connection structure is recorded in frame order, the line segment set structure and the node binding index in the jump segment are obtained, and a multi-segment line segment construction atlas is generated; An atlas is constructed based on the multiple line segments, the number of line segments connected to the coordinate points in the paragraph graph structure is counted and a connection number index table is established according to the node position, the number of connecting edges of each node in the graph structure is counted as the connectivity of the node, and the connectivity of each node is summed to obtain the total connectivity value of the graph structure. Combined with the graph convolutional network, the connectivity of each node is used as the input feature value to model the association relationship between the nodes in the graph structure, complete the mapping of structure number and connectivity set organization, and generate an action topology aggregation structure set.

6. The method for intelligently capturing and processing interactive data of air gestures according to claim 1, characterized in that: The specific steps of outputting a set of identifiable stable action structures are: Based on the action topology aggregation structure set, extract the line segment node index pairs in the graph structure and generate directional coordinate differences according to the start and end coordinate points. Use the directional coordinate differences to convert them into a vector structure in sequence according to the frame number. All vector sets are assigned to corresponding index positions according to the segment number and a segment mapping structure is established to generate a directional vector set within the structure. Based on the direction vector set within the structure, the angles between consecutive vectors in the vector set are calculated and a sequence of angle values is generated. Two consecutive vectors are paired in the form of a ratio of the angle change amplitude to the length difference of the corresponding line segments and the corresponding serial number positions are recorded. The angle change and length ratio sequences are sorted by segment number to generate a sequence of direction change and length ratio. Based on the direction change and length ratio sequence, the slope difference between adjacent elements in the sequence is calculated, the mean of continuous slope changes is counted in the fixed window traversal interval, the fluctuation stable segment is extracted by mean sorting, and a stable segment number structure table is established to generate a recognizable stable action structure set.

7. A system for intelligently capturing and processing data of gestures in the air, applied to the method for intelligently capturing and processing data of gestures in the air according to any one of claims 1 to 6, characterized in that: The system comprises: The motion monitoring module extracts the coordinates of the wrist, palm, and fingertips frame by frame based on the changes in the coordinates of the three points on the time axis in consecutive frame images. It calculates the angle increment and Euclidean distance difference between the lines formed by the three points in adjacent frames. It records the frame numbers with inter-frame distance differences greater than a preset range and maps them to the corresponding time axis positions, generating a sequence of motion direction offset paragraph markers. Trajectory segmentation module: Based on the action direction offset paragraph mark sequence, the three-point coordinate data in the corresponding frame image is extracted according to the paragraph frame number interval, and a paragraph trajectory coordinate set is sequentially constructed. After the trajectory is divided into segments with equal intervals, the distance between the point pairs is calculated to form a distance sequence. The distance values of each trajectory segment are accumulated. In combination with the dynamic time warping method, the trajectory segments are time-aligned with the reference trajectory and the total distance data is generated to establish a list of determinate action trajectory paths; Density screening module: Based on the list of identifiable motion trajectory paths, it extracts the coordinate points of each trajectory segment arranged in chronological order and constructs a set of direction vectors. It calculates the total amount of angle change between adjacent vectors and calculates the angle change density per frame based on the number of frames. It removes segments with a total density below the set jump threshold, retains the high-density segment numbers, and outputs a high-change direction jump segment index group. Structure construction module: Based on the high-variability direction jump segment index group, the coordinate points of the wrist, palm, and fingertips in the corresponding segment are extracted, the Euclidean distance is calculated for each pair of points, and edges are constructed based on the connection relationship. An undirected graph structure is established based on the coordinate points, with the coordinate points set as nodes and the edges representing the connection relationship between the two points. The number of edges of each node is counted and the total connectivity is calculated. A graph convolutional network is introduced, and the connection relationship between nodes is used as input to model the graph structure, generating a set of action topology aggregation structures; Action recognition module: Based on the action topology aggregation structure set, the direction vector set corresponding to each line segment in the graph structure is extracted, the angle change amplitude between adjacent direction vectors and the ratio of the corresponding line segment length are calculated, and a direction change and length ratio sequence is constructed. The continuous slope fluctuation mean of the sequence is calculated within a fixed sliding window, and the stable fluctuation segment is identified by mean sorting. The stable segment number is extracted and a structure index is established, and a set of identifiable stable action structures is output.

Images