A gesture recognition method and system based on video stream analysis

Abstract

This invention belongs to the field of computer vision and human-computer interaction technology, and relates to a gesture recognition method and system based on video stream analysis. The method includes: acquiring video capture device configuration and video stream frame images, writing frame timestamps and sampling at fixed intervals, extracting continuous keyframes and binding keyframe indices to generate a keyframe index structure; performing size normalization and contrast enhancement to obtain an effective gesture image content structure; assembling training samples and training dual models to construct a dual-model aligned result structure; performing multi-frame gesture category consistency comparison, multi-frame pose description consistency comparison, and average confidence threshold comparison to generate gesture judgment results, and constructing a loop control configuration structure. This invention effectively improves the accuracy and robustness of gesture recognition through multi-frame consistency comparison and dynamic loop control mechanisms, reduces the impact of environmental noise and instantaneous errors, and achieves efficient adaptive gesture recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human-computer interaction technology, and in particular relates to a gesture recognition method and system based on video stream analysis. Background Technology

[0002] Currently, gesture recognition technology has been widely applied in smart homes, virtual reality, augmented reality, and in-vehicle systems, becoming an important means of human-computer interaction. Traditional gesture recognition methods typically rely on specific sensors, such as depth cameras or data gloves. While these devices offer high accuracy, they are expensive and have significant limitations in use, hindering widespread adoption. Gesture recognition technology based on ordinary camera (RGB) video streams has attracted widespread attention due to its low cost and ease of integration (e.g., into mobile phones and laptops). However, this technology faces numerous challenges in practical applications. First, real-world background environments are complex and varied, and gestures may occupy a small proportion of the image, easily affected by changes in lighting, partial occupancy, and other factors. Second, gestures themselves possess high flexibility and similarity; different gestures may differ only slightly (e.g., the numbers "1" and "2"), and the same gesture performed by different users exhibits individual differences, all contributing to the difficulty of recognition. Existing video stream-based recognition methods mostly employ a single deep learning model (e.g., CNN) to directly classify images end-to-end. While this method performs well on specific, clean datasets, it is prone to misclassification in practical applications. For example, an action that is similar to but unrelated to the target gesture shape, or a static, non-hand object that coincidentally forms the shape of a gesture, may be misidentified by the model as a valid gesture, resulting in poor system robustness and a poor interactive experience.

[0003] Furthermore, in the field of computer vision technology, existing solutions for gesture recognition based on video acquisition device configuration and video stream frame images typically revolve around steps such as frame timestamp writing, fixed-interval sampling, extraction of three consecutive keyframes, keyframe index binding processing, hand region localization processing, hand region cropping and expansion, size normalization and contrast enhancement operations, training sample assembly, and dual-model training. These solutions suffer from limitations such as incomplete association records between the keyframe index structure and video stream frame images, unclear field connections between invalid image content sets and keypoint annotation datasets in the training sample assembly stage, and inconsistent recording standards in the dual-model alignment result structure during frame alignment and group summarization stages. Existing methods often rely on a single path to output gesture categories or keypoint coordinates. Under complex backgrounds, varying lighting conditions, and interference from irrelevant actions, inconsistencies arise in the criteria used for comparing the consistency of gesture categories across three frames and the consistency of pose descriptions across three frames. The average confidence threshold comparison process and the triggering conditions for gesture determination results are also difficult to maintain consistency. Furthermore, the write-back link of the loop control configuration structure lacks index records consistent with the keyframe index structure, making it difficult to maintain a continuous and closed call relationship between the keyframe index structure, the effective gesture image content structure, and the dual-model alignment result structure. For the joint processing of video acquisition device configuration and video stream frame images in the sampling, alignment, determination, control, and recording stages, existing technologies generally lack a comprehensive data structure and call path covering the writing of frame timestamps to the write-back of the loop control configuration structure. This makes it difficult to complete the state recording from hand region localization processing to training sample assembly and dual-model training within the same processing link, and to establish a consistent closed-loop process between gesture determination results and upstream sampling configuration updates. Consequently, the generation and updating of gesture determination results and the loop control configuration structure lack consistent processing link support. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a gesture recognition method based on video stream analysis, comprising:

[0005] Acquire video capture device configuration and video stream frame images, perform frame timestamp writing, fixed-interval sampling and extraction of three consecutive keyframes, keyframe index binding processing, and generate keyframe index structure;

[0006] Based on the keyframe index structure, YOLOv5s hand region localization processing is performed, followed by hand region cropping and expansion, size normalization, and contrast enhancement processing to obtain an effective gesture image content structure.

[0007] Based on the effective gesture image content structure, training sample assembly and dual-model training operations are performed, and three-frame parallel inference and pose description generation processing are executed to construct the dual-model aligned result structure.

[0008] Based on the dual-model alignment result structure, the system performs three-frame gesture category consistency comparison, three-frame pose description consistency comparison, and average confidence threshold comparison to generate gesture determination results and construct a loop control configuration structure.

[0009] Furthermore, the process of writing frame timestamps also includes:

[0010] The frame timestamp writing process includes providing time stamps via a monotonically increasing system clock or acquisition hardware clock. When both hardware clock and system clock sources exist, a clock alignment table is used to complete time source mapping, converting the hardware timestamps into a unified system timestamp. Simultaneously, a video frame sequence is generated as an intermediate product. The video frame sequence contains a set of frame records enqueued in ascending order of frame number. The set of frame records includes frame content, frame number, frame timestamp, acquisition channel identifier, and anomaly flag. A circular buffer is used to maintain the most recent consecutive frame window.

[0011] Furthermore, the fixed-interval sampling process also includes:

[0012] Fixed-interval sampling processing involves selecting a sampling start point from a candidate frame window in the video frame sequence according to the sampling interval. The sampling interval is given by a fixed-interval sampling configuration, which includes a sampling interval value, a start offset, and an allowed number of frame skips, as well as a sampling start point reset condition. The sampling start point reset condition refers to triggering a reset operation when a switching of the acquisition channel identifier is detected, a clock back-back flag is detected during continuity checks, or an abnormal flag appears consecutively.

[0013] Furthermore, the process of extracting three consecutive keyframes and binding keyframe indexes also includes:

[0014] The three consecutive keyframe extraction process involves reading three consecutive frame records from the sampling start point, and the three frame records must have the same acquisition channel identifier and consecutive frame numbers. If a missing frame is encountered, the next frame is searched to fill the missing frame according to the allowed number of frame skips. Keyframe index binding is performed to generate a keyframe index structure, which includes references to the three keyframe images, corresponding frame numbers, corresponding frame timestamps, acquisition channel identifiers, anomaly marker summaries, and incomplete markers.

[0015] Furthermore, the process of performing YOLOv5s hand region localization processing also includes:

[0016] The hand region localization process uses the YOLOv5s object detection model to process three keyframe images frame by frame. The YOLOv5s object detection model includes a model weight file, a class mapping table, input size configuration, and inference threshold configuration. The inference threshold configuration includes a candidate box confidence threshold and an overlap suppression threshold. The hand region localization process includes feeding the three keyframe images frame by frame into the inference pipeline to perform pixel format alignment and scale adjustment, outputting a candidate bounding box set, and performing overlap suppression processing on the candidate bounding box set to form a valid bounding box set. When the valid bounding box set is empty, a non-detected flag is written and the current round of processing ends. When there are multiple valid bounding boxes, they are sorted by bounding box confidence and the main bounding box is selected based on the bounding box area constraint.

[0017] Furthermore, the process of cropping and expanding the hand region, normalizing its size, and enhancing its contrast to obtain an effective gesture image content structure also includes:

[0018] The hand region cropping and expansion operation involves expanding the width and height of the bounding box outwards from the main bounding box. The expansion ratio is defined as expanding the width and height of the bounding box by 10% each. After expansion, boundary cropping correction is performed. For frames with displacement or scale anomaly markers, cropping consistency correction is also performed, smoothly blending the coordinates of the expanded bounding box of that frame with the coordinates of the expanded bounding boxes of adjacent frames. Next, size normalization and contrast enhancement operations are performed. The size normalization operation involves scaling the cropped gesture image to a uniform spatial size of 384 pixels in both width and height, using a scaling strategy that maintains the aspect ratio. When the aspect ratio is inconsistent, edge padding is applied to the longer side. The contrast enhancement operation involves performing block histogram equalization on the size normalized image to obtain local contrast improvement. Finally, an effective gesture image content structure is generated. The effective gesture image content structure includes three normalized gesture images, three image quality markers, corresponding frame numbers, and corresponding frame timestamps. The image quality markers include sharpness description, brightness description, motion blur description, and anomaly marker summary.

[0019] Furthermore, the process of performing three-frame parallel inference and pose description generation to construct the dual-model aligned result structure also includes:

[0020] The three-frame parallel inference processing involves feeding three normalized gesture images into the multimodal classification model inference pipeline and the key point recognition model inference pipeline in parallel, and outputting the three-frame gesture category, three-frame confidence and three-frame key point coordinates.

[0021] The pose description generation process includes extracting fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features from the three frames of joint coordinates, and performing denoising and smoothing processing on the joint coordinates; finally, a dual-model aligned result structure is constructed.

[0022] The dual-model alignment result structure includes three frames of gesture categories, three frames of confidence, three frames of joint coordinates, three frames of posture descriptions and corresponding frame numbers, and corresponding frame timestamps.

[0023] Furthermore, the process of comparing the consistency of gesture categories across three frames also includes:

[0024] The three-frame gesture category consistency comparison process includes comparing the gesture categories of the three frames in the same group, comparing whether the categories of adjacent frames are the same in order of frame number and checking whether all three frame categories are the same, outputting a consistency flag, and when they are consistent, the common category is used as a candidate gesture category, and when they are inconsistent, a category conflict flag is written.

[0025] Furthermore, the process of comparing the consistency of the three-frame pose descriptions with the average confidence threshold also includes:

[0026] The three-frame pose description consistency comparison process includes extracting fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features from the three-frame pose descriptions, and performing intra-frame consistency checks and cross-frame consistency checks on each type of sub-feature set. The intra-frame consistency check includes verifying whether the joint index is complete, and the cross-frame consistency check includes judging whether the feature changes meet the constraints through a preset threshold set, and outputting a pose consistency flag.

[0027] The average confidence threshold comparison process involves performing validity screening on the confidence of three frames and calculating the average confidence, then comparing the average confidence with a preset confidence threshold.

[0028] Furthermore, a gesture recognition system based on video stream analysis, applied to any of the methods described above, includes:

[0029] The video stream acquisition and keyframe extraction module is used to acquire the video acquisition device configuration and video stream frame images, write frame timestamps, sample at fixed intervals and extract three consecutive keyframes, bind keyframe indexes and generate keyframe index structures.

[0030] The hand region localization and effective gesture image generation module is used to receive the keyframe index structure and perform YOLOv5s hand region localization processing, perform hand region cropping and expansion, size normalization and contrast enhancement operations to obtain the effective gesture image content structure.

[0031] The dual-model training and parallel inference module is used to receive the effective gesture image content structure and access the invalid image content set and key point annotation dataset, perform training sample assembly and dual-model training operations, and execute three-frame parallel inference and pose description generation processing to construct the dual-model aligned result structure.

[0032] The consistency joint judgment and loop control module is used to receive the dual-model alignment result structure, perform three-frame gesture category consistency comparison, three-frame pose description consistency comparison and average confidence threshold comparison processing, generate gesture judgment results and construct loop control configuration structure, and write back the loop control configuration structure to the video stream acquisition and key frame capture module.

[0033] The key innovations of this invention include:

[0034] (1) A key frame index structure is formed by writing frame timestamps, sampling at fixed intervals and extracting three consecutive key frames, and combining key frame index binding processing, and serves as a unified input object for subsequent processing.

[0035] (2) Establish an integrated link between detection and preprocessing around the keyframe index structure and the effective gesture image content structure. Through YOLOv5s hand region localization processing and combined with hand region cropping and expansion, size normalization and contrast enhancement operations, the keyframe index structure is converted into the effective gesture image content structure.

[0036] (3) The training and inference of effective gesture image content structure are integrated. Training samples are assembled and dual-model training is performed by accessing the invalid image content set and the key point annotation dataset. Three-frame parallel inference and posture description generation are performed to construct the dual-model alignment result structure. Then, the gesture judgment result is generated by comparing the consistency of the three-frame gesture categories, the consistency of the three-frame posture description, and the average confidence threshold. A loop control configuration structure is constructed.

[0037] The following are its main beneficial effects:

[0038] (1) In view of the problem that video stream frame images lack consistent index carrying after fixed interval sampling and key frame extraction in the existing scheme, the present invention solidifies the frame timestamp written with the association record of three consecutive key frame extractions into a key frame index structure through key frame index binding processing, so that the key frame index structure maintains the same input caliber and traceable association relationship in the subsequent YOLOv5s hand region localization processing and dual model alignment result structure generation process.

[0039] (2) In view of the problem that the connection between hand region localization processing and hand region cropping expansion, size normalization and contrast enhancement operations in the existing scheme is unstable and the input objects are difficult to unify, the present invention uses the key frame index structure as a unified entry point and completes YOLOv5s hand region localization processing and image preprocessing operations in the same module link, outputting an effective gesture image content structure, so that subsequent training sample assembly and three-frame parallel inference can obtain consistent image content input and reference to the same source data.

[0040] (3) In view of the problem that the lack of an alignable result organization between the training sample assembly, dual-model training operation and inference judgment link in the existing scheme leads to the dispersion of judgment basis, the present invention forms a unified data organization for training sample assembly and dual-model training operation under the condition of accessing invalid image content set and key point annotation dataset, and constructs a dual-model alignment result structure in the three-frame parallel inference and pose description generation processing stage, so that the three-frame gesture category consistency comparison, three-frame pose description consistency comparison and average confidence threshold comparison processing share the same alignment result input. At the same time, the loop control configuration structure implements back-write update of the key frame index structure generation link, thereby forming a consistent closed-loop process of judgment and update. Attached Figure Description

[0041] Figure 1 An overall result block diagram of a gesture recognition system based on video stream analysis provided in this application embodiment;

[0042] Figure 2 A flowchart illustrating a gesture recognition method based on video stream analysis provided in an embodiment of this application;

[0043] Figure 3 A flowchart of a dual-model joint decision-making process is provided for an embodiment of this application;

[0044] Figure 4 This is a structural block diagram of a gesture recognition system based on video stream analysis provided in an embodiment of this application. Detailed Implementation

[0045] In one embodiment, Figure 1 This application provides an overall result block diagram of a gesture recognition system based on video stream analysis. The workflow of this system can be summarized as follows: Figure 1 The five stages shown are, in order: video stream acquisition, keyframe extraction, image preprocessing, dual-model judgment, and gesture output. Specifically, the system first captures the original video stream using a video acquisition device (such as a camera); then, based on a fixed-interval sampling strategy, it extracts multiple consecutive frames (such as three consecutive frames) as a keyframe group; furthermore, it performs preprocessing operations such as hand region localization, cropping, normalization, and enhancement on each keyframe image to obtain a regular and valid gesture image; then, it inputs the preprocessed multi-frame images in parallel into a trained multimodal classification model and a keypoint recognition model for inference, obtaining multi-dimensional information such as gesture category, confidence level, and keypoint coordinates; finally, by comparing the consistency and judging the confidence level of the inference results of multiple frames, it generates the final gesture judgment result or abandonment signal, thus completing a complete and adaptive closed-loop processing from the original video input to a highly reliable gesture recognition output.

[0046] A preferred embodiment, referring to Figure 2This is a flowchart illustrating a gesture recognition method based on video stream analysis provided in an embodiment of the present invention. The flowchart may include at least steps S100-S400:

[0047] S100: Obtain the video acquisition device configuration and video stream frame images, perform frame timestamp writing, fixed-interval sampling and extraction of three consecutive key frames, key frame index binding processing, and generate a key frame index structure.

[0048] S200. Based on the keyframe index structure, perform YOLOv5s hand region localization processing, perform hand region cropping and expansion, size normalization and contrast enhancement processing to obtain an effective gesture image content structure.

[0049] S300: Based on the effective gesture image content structure, perform training sample assembly and dual-model training operations, and execute three-frame parallel inference and pose description generation processing to construct the dual-model aligned result structure.

[0050] S400, based on the dual-model alignment result structure, performs three-frame gesture category consistency comparison, three-frame pose description consistency comparison and average confidence threshold comparison processing to generate gesture judgment results, and constructs a loop control configuration structure.

[0051] S100: Obtain the video acquisition device configuration and video stream frame images, perform frame timestamp writing, fixed-interval sampling and extraction of three consecutive key frames, key frame index binding processing, and generate a key frame index structure.

[0052] Specifically, this step establishes a processing link starting with two types of inputs: video acquisition device configuration and video stream frame images. The video acquisition device configuration refers to a set of parameters including camera hardware identifier, installation orientation, resolution, frame rate, exposure and gain, white balance, focus mode, data transmission channel, and buffer depth. This parameter set is obtained from the device's factory calibration information, system initialization configuration file, or written through the interactive configuration interface. The frame rate refers to the number of frames acquired per unit time; this step configures the frame rate to 25 frames per second or 30 frames per second and records the current configuration source when writing the configuration. Further, this step performs a validity check on the video acquisition device configuration. This validity check includes resolution matching with device capabilities, frame rate matching with data transmission channel bandwidth, and exposure and gain value boundary checks. When a check fails, this step records the reason for the failure and rolls back to the previous effective configuration, while also writing the rollback event to the running log. When the system is in a loop operation state, it can also read the fixed-interval sampling configuration from the loop control configuration structure output by S400 and fill it back into the video acquisition device configuration of this step, serving as the sampling basis for keyframe capture in this round.

[0053] Video stream frame images refer to the continuous frame-by-frame image data output by an RGB (Red, Green, Blue) camera. The frame-by-frame image data includes frame content, frame number, image format identifier, and acquisition channel identifier. During the access process, the acquisition end establishes an acquisition channel through the device driver interface, allocates a frame buffer queue, and starts the acquisition thread. When each frame arrival event is triggered, the acquisition end reads the frame content from the frame buffer and performs image format unification processing, organizing the frame content into a unified pixel format and writing the image format identifier and acquisition channel identifier to form a frame image record. Furthermore, when a camera disconnection, acquisition channel abnormality, interface call timeout, or frame buffer emptiness is detected, the acquisition end writes an exception flag and records the time of the exception, pausing keyframe capture for this round until subsequent valid frames arrive.

[0054] Further, this step performs frame timestamp writing processing on the video stream frame images to generate a video frame sequence as an intermediate product. A frame timestamp refers to a time stamp corresponding to a single frame image. This timestamp is provided by a monotonically increasing system clock or acquisition hardware clock, and the time source type is recorded synchronously during writing. When both hardware clock and system clock sources exist, this step completes time source mapping through a clock alignment table, converting the hardware timestamp into a unified system timestamp and writing it into the frame image record. Simultaneously, the mapping version number is recorded for traceability. Specifically, for each frame image record, this step reads the timestamp and writes the frame timestamp after switching the frame buffer to a readable state, generates a frame sequence number, and binds it to the acquisition channel identifier, forming a frame record set containing frame content, frame sequence number, frame timestamp, acquisition channel identifier, and anomaly flag. The frame record set is enqueued in ascending order of frame sequence number, and a circular buffer is used to maintain the most recent consecutive frame window, thus forming a video frame sequence. Further, this step performs a continuity check on the frame record set. This continuity check includes frame number increment checking, frame timestamp monotonic checking, and acquisition channel identifier consistency checking. When a frame number jump is detected, this step writes a missing frame flag and records the start and end frame numbers of the missing frame interval. When a frame timestamp rollback is detected, this step writes a clock rollback flag and records the time difference description of the rollback magnitude. When a missing frame flag or clock rollback flag occurs, this step retains the corresponding frame record in the video frame sequence for subsequent skipping or delay processing during keyframe capture. When the circular buffer reaches its limit, this step evicts the earliest frame record using a first-in-first-out (FIFO) strategy and writes the frame number range of the evicted frame record into the evictation log. When the anomaly flag reaches a preset number within a continuous window, this step triggers an acquisition channel reconnection operation, updates the acquisition channel identifier or records reconnection failure information after the reconnection result is returned, and continues to append frame records.

[0055] After obtaining the video frame sequence, this step performs fixed-interval sampling and extraction of three consecutive keyframes, and binds the keyframe index to output the keyframe index structure. Fixed-interval sampling refers to selecting a sampling starting point from the candidate frame window according to the sampling interval in the video frame sequence. The sampling interval is given by the fixed-interval sampling configuration, which includes the sampling interval value, starting offset, and allowed frame skipping count, as well as a sampling starting point reset condition. The sampling starting point reset condition is triggered when a switching of the acquisition channel identifier, a clock back-up flag during continuity checking, or consecutive occurrences of anomaly flags are detected. Extraction of three consecutive keyframes refers to continuously reading three frame records from the sampling starting point. The three frame records must have the same acquisition channel identifier and consecutive frame numbers. If a missing frame flag is encountered, the next frame is searched for according to the allowed frame skipping count to complete the extraction; otherwise, the current group of three keyframe images is marked as incomplete and delayed until the next round of extraction. Specifically, this step locates the sampling start frame number in the circular buffer, reads three frame records sequentially and combines them into three key frame images, and binds the frame number, frame timestamp and acquisition channel identifier of the three key frame images to generate a key frame index structure. The key frame index structure includes references to the three key frame images, corresponding frame numbers, corresponding frame timestamps, acquisition channel identifiers, anomaly marker summary and incomplete marker.

[0056] The keyframe index structure is recorded as the output field name of this step and passed to the hand region localization and effective gesture image generation link as the input keyframe index structure of the subsequent S200. The subsequent S200 performs hand region localization processing based on the keyframe index structure and outputs the effective gesture image content structure. The effective gesture image content structure is used by S300 to perform training sample assembly and dual-model training operations. After generating the loop control configuration structure, S400 writes back the fixed interval sampling configuration to this step to complete the loop connection across the main steps.

[0057] Summary of the technical effects of this step: This step completes the writing of time stamps for video stream frame images, fixed-interval sampling, and organization of three consecutive keyframes, generating a keyframe index structure and connecting it to the subsequent main processing steps.

[0058] S200. Based on the keyframe index structure, perform YOLOv5s hand region localization processing, perform hand region cropping and expansion, size normalization and contrast enhancement processing to obtain an effective gesture image content structure.

[0059] Specifically, the input source for this step is the keyframe index structure output by S100. This keyframe index structure includes references to three keyframe images, corresponding frame numbers, corresponding frame timestamps, acquisition channel identifiers, anomaly marker summaries, and incomplete markers. After accessing the keyframe index structure, this step first performs an input integrity check. The check includes verifying the existence of the three keyframe image references, the continuity of the corresponding frame numbers, the consistency of the acquisition channel identifiers, and whether the anomaly marker summary triggers a skip condition. When the incomplete marker is true or the anomaly marker summary triggers a skip condition, this step records the frame number range and skip reason for this keyframe group, marks the keyframe group as an invalid pending record, and requests the upstream to send the next set of keyframe index structures to this step. These invalid pending records are used for subsequent invalid image content set assembly link calls during the training phase. After completing the input check, this step enters the hand region localization processing link. This hand region localization processing link is executed frame-by-frame for the three keyframe images, and the frame-by-frame outputs are bound together for subsequent cropping, expansion, size normalization, and contrast enhancement operations.

[0060] Further, this step employs the YOLOv5s object detection model to perform hand region localization. YOLO stands for You Only Look Once. The YOLOv5s object detection model consists of a model weight file, a class mapping table, input size configuration, and inference threshold configuration. The class mapping table defines the correspondence between hand target category identifiers and category indices. The inference threshold configuration includes candidate box confidence thresholds and overlap suppression thresholds. During runtime, this step feeds three keyframe images into the inference pipeline frame by frame. Before the images are input, the inference pipeline performs pixel format alignment and scale adjustment, and writes the adjusted image tensors to the inference buffer. After inference, a candidate bounding box set is output. Each item in the candidate bounding box set includes bounding box coordinates and a bounding box confidence score. Bounding box coordinates are defined as the pixel positions of the top-left and bottom-right corners, and the bounding box confidence score describes the model's confidence that the candidate bounding box belongs to the hand target category. This step performs overlap suppression processing on the candidate bounding box set. Overlap suppression processing removes highly overlapping candidate boxes based on the overlap suppression threshold, retaining candidate boxes with higher confidence scores to form a valid bounding box set. When the set of valid bounding boxes is empty, this step writes the corresponding frame into an undetected flag and records the inference threshold configuration, frame number, and acquisition channel identifier. At the same time, the three keyframe images in this group are marked as invalid records to be processed and the current round of processing ends, and the process moves to the next set of keyframe index structures. When there are multiple valid bounding boxes, this step sorts them by bounding box confidence and selects the main bounding box in combination with the bounding box area constraint. The bounding box area constraint is defined as the candidate box area must be within the preset minimum area and maximum area range. Those that exceed the range are written into a boundary anomaly flag and removed. The main bounding box is used for subsequent hand area cropping and expansion operations.

[0061] After obtaining the frame-by-frame main bounding boxes, this step performs group consistency constraint processing on the three main bounding boxes and outputs the bounding box coordinates and bounding box confidence scores as intermediate products. The group consistency constraint processing includes bounding box center point displacement constraints and bounding box scale change constraints. The bounding box center point displacement constraint is defined as writing a displacement anomaly flag when the displacement of the center point of the main bounding box in adjacent frames exceeds a threshold. The bounding box scale change constraint is defined as writing a scale anomaly flag when the area change of the main bounding box in adjacent frames exceeds a threshold. When a displacement anomaly flag or a scale anomaly flag appears, this step does not terminate the localization link. Instead, it binds the anomaly flag summary to the bounding box coordinates and bounding box confidence scores of the corresponding frame for subsequent boundary cropping correction and recording in the cropping and expansion stage. Furthermore, when the main bounding box coordinates exceed the original frame image boundary, this step performs boundary cropping correction, cropping the excess portion to the effective image range and recording the coordinate difference before and after correction, forming a traceable localization record. At this point, this step has obtained the bounding box coordinates and bounding box confidence, and bound them to the three keyframe image references, frame numbers, and frame timestamps in the same frame, serving as the input source for hand region cropping expansion and size normalization.

[0062] Further, this step performs a hand region cropping and expansion operation. This expansion refers to extending the width and height of the bounding box outwards from the main bounding box. The expansion ratio is defined as expanding the width and height of the bounding box by 10% each. After expansion, a boundary cropping correction is performed again to prevent out-of-bounds errors. Specifically, this step calculates the width and height of the bounding box from its coordinates, calculates the expansion margin according to the expansion ratio, and generates the expanded bounding box coordinates. Subsequently, the hand image region is cropped from the original frame image referenced by the three keyframe images according to the expanded bounding box coordinates, resulting in a cropped gesture image. For frames with displacement or scale anomaly markers, this step also performs crop consistency correction processing. This process smoothly merges the expanded bounding box coordinates of this frame with those of adjacent frames, and records the fusion weight and the merged coordinates. The fusion weight is determined by the bounding box confidence and anomaly marker summary; frames with higher confidence and fewer anomaly markers receive higher weights. If the cropping gesture image is empty or has an abnormal size, this step writes a cropping failure mark and records the reason for the failure, ends the current round of processing, marks the three keyframe images in this group as invalid records to be processed, and moves on to the next group of keyframe index structures.

[0063] After obtaining the cropped gesture image, this step performs size normalization and contrast enhancement operations to generate an effective gesture image content structure. Size normalization refers to scaling the cropped gesture image to a uniform spatial size. In this step, the size normalization is configured to be 384 pixels in both width and height, and a scaling strategy that maintains the aspect ratio is adopted. When the aspect ratio of the cropped gesture image is inconsistent with the target size, this step first scales it to the target size along the shorter side, and then fills the edges along the longer side. The fill value is obtained by statistical analysis of neighboring pixels, and the range of the filled area and the statistical summary of the fill value are recorded. The record is used for sample quality tracking in the subsequent training sample assembly stage. Contrast enhancement refers to performing brightness and contrast adjustments on the size-normalized image. This step uses block histogram equalization to obtain local contrast improvement and sets upper and lower limits for the enhancement intensity. If overexposure or underexposure occurs after enhancement, this step reduces the enhancement intensity and performs enhancement again. If the abnormality still occurs, an image enhancement anomaly marker is written, and the original size-normalized image is retained as a replacement output. Further, this step generates an image quality tag for each frame of the normalized gesture image. The image quality tag is defined as a combined record of a sharpness description, a brightness description, a motion blur description, and an anomaly tag summary. The sharpness description is obtained through image edge gradient statistics, the brightness description is obtained through grayscale mean statistics, the motion blur description is obtained through adjacent frame difference statistics, and the anomaly tag summary comes from the anomaly tags in the aforementioned positioning and cropping stages. The normalized gesture image and the image quality tag are bound together in the same frame and assembled with the three frame sequence numbers and three frame timestamps to form a valid gesture image content structure.

[0064] The output of this step is a valid gesture image content structure. This valid gesture image content structure is recorded as an output field name at the end of this step and is used as an input item of S300 to be sent to the subsequent training sample assembly and dual-model training operation. At the same time, the invalid unprocessed records formed in this step are incorporated into the invalid image content set in subsequent stages for sample construction link. The keyframe index structure on which this step depends comes from S100. The cross-main step connection is continuously transmitted during runtime by field binding records and frame sequence number alignment records. After the loop control configuration structure generated by S400 writes back the fixed interval sampling configuration, the upstream S100 updates the sampling rhythm and re-outputs the keyframe index structure to enter this step, thereby forming a loop operation process.

[0065] Summary of the technical effects of this step: This step completes the localization, cropping and expansion, size normalization and contrast enhancement of the hand region in three keyframe images, generates an effective gesture image content structure and connects to the subsequent main processing steps.

[0066] S300: Based on the effective gesture image content structure, perform training sample assembly and dual-model training operations, and execute three-frame parallel inference and pose description generation processing to construct the dual-model aligned result structure.

[0067] Specifically, this step takes the valid gesture image content structure output by S200 as the main input, and simultaneously inputs the invalid image content set and keypoint annotation dataset as training and calibration inputs. The valid gesture image content structure refers to a frame-bound record consisting of three normalized gesture images, three image quality markers, corresponding frame numbers, and corresponding frame timestamps. The three normalized gesture images are hand image regions that have undergone cropping, expansion, size normalization, and contrast enhancement. The image quality markers include sharpness descriptions, brightness descriptions, motion blur descriptions, and anomaly marker summaries. The invalid image content set refers to a summary of invalid records to be processed under conditions such as incomplete keyframe groups, undetected hand region localization, cropping failure, or abnormal image enhancement. This summary includes at least an invalid frame image reference, an invalidity reason identifier, a corresponding frame number, and a corresponding frame timestamp. The keypoint annotation dataset refers to a collection of annotation data for hand image regions. This annotation data includes at least hand image references, coordinate annotations of twenty-one human hand keypoints, and annotation consistency markers. The coordinate annotations of the twenty-one human hand keypoints record the pixel coordinates of the wrist point, finger root point, finger node, and fingertip point in keypoint index order, and are bound to annotator identifiers or annotation source identifiers. Upon entering this step, consistency checks are first performed on the three types of inputs. The checks include whether the frame sequence number and frame timestamp are aligned, whether the image reference is readable, whether the annotation coordinates fall within the image boundaries, and whether the annotation consistency markers trigger the removal conditions. When unreadable image references or out-of-bounds annotation coordinates exist, this step records the abnormal entries and removes the corresponding samples. Simultaneously, the reason for removal is written into the training data cleanup record to prevent unparseable data from appearing in subsequent training sample assembly.

[0068] Further, this step performs training sample assembly processing to generate model training samples and complete sample versioning records. Specifically, three normalized gesture images and three image quality markers are extracted from the valid gesture image content structure to form valid sample entries; invalid frame image references are extracted from the invalid image content set and the frame image content is read to form invalid sample entries; hand image references and twenty-one human hand key point coordinate annotations are extracted from the key point annotation dataset to form key point sample entries. The model training samples consist of two types of sub-samples: one type is multimodal training samples, which at least contain gesture image content and gesture category labels or invalid category labels. The gesture category labels are derived from a gesture category mapping table, which maps gesture category names to category indices and records mapping version numbers; the other type is key point training samples, which at least contain gesture image content and twenty-one human hand key point coordinate annotations. Furthermore, this step performs sample balancing on the model training samples. This balancing process includes counting the number of samples by gesture category label, counting the number of samples by invalid category label, and counting the quality distribution by image quality label. Based on the statistical results, resampling or undersampling is performed. When the number of samples in a certain category falls below a threshold, this step records the insufficient number of samples in that category and reduces the participation ratio of that category in the current training round, while retaining that category for subsequent training rounds. Further, this step performs a partitioning process on the model training samples, dividing them into a training subset and a validation subset, and recording the partitioning random seed identifier and partitioning version number. These records are written into the training process log for parameter tracking in the subsequent inference phase.

[0069] Figure 3 A flowchart of a dual-model joint decision-making process is provided for embodiments of this application; as follows: Figure 3 As shown, the dual-model decision-making stage constructs a parallel multi-information-flow fusion architecture. The specific processing flow is as follows: After three frames of images are input, the hand region is first located and cropped using the YOLOv5 segmentation step. Then, the processing flow is divided into two parallel branches: the left branch inputs the image into the SigmaPipe classification gesture recognition model, outputting the gesture category and its corresponding confidence score; the right branch inputs the image into the MediaPipe model, outputting the coordinates of 21 hand joint points. The outputs of the two branches are then combined and enter the consistency + confidence score judgment stage, where the classification results and pose features of the three frames are jointly analyzed, ultimately deciding and outputting the valid gesture. This process clearly demonstrates the information transmission and decision-making path from the original frames to the final judgment, reflecting the core design of dual-model information complementarity and joint decision-making.

[0070] Specifically, after constructing the model training samples, this step performs dual-model training, generating dual-model parameters and writing them into the model version record. Dual-model training refers to updating the parameters of both the multimodal classification model and the keypoint recognition model. The multimodal classification model outputs the gesture category and confidence score, while the keypoint recognition model outputs the coordinates of twenty-one key points on the human hand and supports the generation of posture descriptions. Specifically, the multimodal classification model uses the siglip2-so400m-patch14-384 model as the base model, which is a pre-trained visual semantic model. In this step, the base model parameters are loaded during the training phase, and some low-level feature extraction layers are fixed. Classification head parameters are updated for the category set defined in the gesture category mapping table. The training input consists of the gesture image content and corresponding labels of the multimodal training samples. During the training process, the loss convergence information, sample batch identifiers, and abnormal batch markers for each iteration are recorded. When gradient anomalies or loss mutations occur, this step records the abnormal iteration number and reverts to the previous stable model parameters. The keypoint recognition model is trained or calibrated using the MediaPipe Hands model framework. MediaPipe is the name of Google's open-source multimedia processing framework, and Hands is the name of the hand keypoint recognition submodule. This step loads the initial parameters of the keypoint recognition model and inputs keypoint training samples during the training phase, performing regression parameter updates for the coordinate annotations of twenty-one human hand keypoints. When the annotation consistency flag triggers the elimination condition, this step skips the corresponding sample and records the reason for skipping. After the dual-model training is complete, this step writes the multimodal classification model parameters and the keypoint recognition model parameters into the dual-model parameters, and binds the dual-model parameters to the model version identifier, training data version number, gesture category mapping table version number, and training process record index, forming a traceable model version record for subsequent three-frame parallel inference.

[0071] Furthermore, this step performs three-frame parallel inference and pose description generation processing in online or offline verification scenarios to construct a dual-model alignment result structure. Specifically, this step reads three normalized gesture images from the effective gesture image content structure and feeds these three images in parallel into the multimodal classification model inference pipeline and the keypoint recognition model inference pipeline. The multimodal classification model inference outputs three gesture categories and three confidence scores, where the confidence score is a description of the confidence level of the category output and is bound to the corresponding frame number. The keypoint recognition model inference outputs three frames of keypoint coordinates, where the three frames of keypoint coordinates are inference results in the format of twenty-one human hand keypoint coordinate annotations and are bound to the corresponding frame number. When any frame inference result has an empty output or out-of-bounds coordinates, this step writes an inference anomaly flag and records the anomaly frame number. The anomaly frame is marked as a low-confidence entry when participating in subsequent alignment. Subsequently, this step performs pose description generation processing, which involves extracting fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features from the three frames of joint coordinates. The fingertip relative distance feature refers to the set of pixel distances between different fingertip pairs, the finger-to-finger angle feature refers to the set of angles formed by connecting adjacent finger points, and the horizontal displacement feature refers to the set of horizontal coordinate changes of the same fingertip point between adjacent frames. During the generation process, this step first performs noise reduction and smoothing processing on the joint coordinates, then calculates each feature according to the joint index relationship, and binds the feature results to the same frame. Finally, this step aligns the three-frame gesture categories, three-frame confidence scores, three-frame keypoint coordinates, three-frame pose descriptions with their corresponding frame numbers and timestamps, and summarizes them within the group to generate a dual-model alignment result structure. This dual-model alignment result structure is recorded as the output field name of this step and is called by the dual-model alignment result structure of the input item S400 to enter the three-frame gesture category consistency comparison, three-frame pose description consistency comparison, and average confidence threshold comparison processing link. At the same time, the effective gesture image content structure comes from S200, and the keyframe index structure comes from S100 and enters this step after being formed into an effective gesture image content structure by S200. When S400 generates the loop control configuration structure and writes back the upstream sampling configuration, the upstream S100 outputs the keyframe index structure again to enter S200 and form a new effective gesture image content structure, thus forming a continuous running link with this step.

[0072] Summary of the technical effects of this step: This step completes the assembly of training samples, dual-model training, three-frame parallel inference and pose description generation, generates the dual-model aligned result structure and passes it to the subsequent main steps.

[0073] S400: Based on the dual-model alignment result structure, perform three-frame gesture category consistency comparison, three-frame pose description consistency comparison and average confidence threshold comparison to generate gesture judgment results and construct a loop control configuration structure.

[0074] Specifically, this step takes the dual-model alignment result structure output by S300 as the sole main input to the decision chain. The dual-model alignment result structure refers to the same-frame alignment record formed after parallel inference of the same set of three-frame normalized gesture images. The record includes at least three-frame gesture categories, three-frame confidence scores, three-frame key point coordinates, three-frame pose descriptions, and corresponding frame numbers and timestamps. The three-frame gesture categories are the category outputs of the multimodal classification model, the three-frame confidence scores are the credibility descriptions corresponding one-to-one with the three gesture categories, the three-frame key point coordinates are the inference results of the twenty-one human hand key point coordinates output by the key point recognition model, and the three-frame pose descriptions are a combination record of fingertip relative distance features, finger angle features, and horizontal displacement features extracted from the three-frame key point coordinates and bound together in the same frame. After accessing the dual-model alignment result structure, this step first performs input structure verification processing. This processing includes checking the continuity of frame numbers, the monotonicity of frame timestamps, whether there are empty outputs of the three-frame keypoint coordinates, and whether there are missing items in the three-frame pose descriptions. When there are empty outputs, missing items, or discontinuous frame numbers, this step writes an input anomaly marker and records the anomaly frame number and anomaly type. Simultaneously, the three-frame record is marked as a resampling item and transferred to the loop control configuration update link, thereby triggering the upstream to regenerate the keyframe index structure and enter subsequent rounds of processing. After completing the input structure verification, this step enters the consistency comparison link, continuously processing the consistency comparison of the three-frame gesture categories, the consistency comparison of the three-frame pose descriptions, and the comparison with the average confidence threshold, and forming a traceable judgment process record during the processing.

[0075] Further, this step performs a three-frame gesture category consistency comparison, which refers to comparing the three frame gesture categories in the same group and outputting a consistency flag. Specifically, this step reads the three frame gesture categories from the dual-model alignment result structure and compares whether adjacent frame categories are the same in order of frame number, while also comparing whether all three frame categories are the same. If any frame category is missing or marked as unusable by an input anomaly flag, the consistency comparison of that group of three frame gesture categories directly writes an inconsistency flag and records the source of the missing category. This step also binds and records the source of the gesture category output. The binding record includes the multimodal classification model version identifier and the gesture category mapping table version number, which is used for subsequent tracing of the judgment results. Further, when the three-frame gesture category consistency comparison obtains a consistency flag, this step writes the common category among the three frame gesture categories as a candidate gesture category into a candidate entry and binds the candidate entry to the corresponding frame number interval and the corresponding frame timestamp interval. When an inconsistency flag is obtained, this step still retains the original output of the three frame gesture categories and writes a category conflict flag for reference when generating the subsequent loop control configuration structure. The category conflict marker contains a conflict frame sequence number and a conflict category pair, which is used to describe the source of inconsistency in the three frame categories.

[0076] Further, this step performs a three-frame attitude description consistency comparison, which involves comparing the three attitude descriptions within the same group and outputting an attitude consistency flag. Specifically, this step extracts the three-frame attitude descriptions from the dual-model alignment result structure and further decomposes them into three sub-feature sets: fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features. Then, for each sub-feature set, intra-frame consistency checks and cross-frame consistency checks are performed. Intra-frame consistency checks are used to verify whether the correspondence between the attitude description and the three-frame joint coordinates is complete. Specifically, this includes whether the joint indexes are complete, whether the joint indexes referenced in the attitude description exist in the three-frame joint coordinates, and whether there are any unresolvable entries in the feature values. When an unresolvable entry is found, this step writes an attitude missing item flag and records the missing item type, while marking the attitude description of that frame as unusable. Cross-frame consistency checking is used to determine whether the three-frame attitude descriptions meet consistency constraints within a group. Specifically, these constraints include constraints on the relative distance between fingertips, the inter-finger angle, and the horizontal displacement. These constraints are provided by a preset threshold set, which is recorded in the attitude consistency configuration and can be updated by the loop control configuration structure. Further, when an unusable flag exists in the three-frame attitude descriptions, this step directly outputs an attitude inconsistency flag and records the frame number of the unusable flag. When all three frame attitude descriptions are usable, this step calculates the sub-feature differences between adjacent frames according to their frame numbers and compares them with the change constraints in the attitude consistency configuration. If the differences all fall within the allowable range, an attitude consistency flag is output; otherwise, an attitude inconsistency flag is output, and the type of out-of-limit sub-feature and the corresponding frame number range are recorded. The attitude consistency comparison process record, together with the aforementioned category conflict flags, forms a component of the consistency comparison result and is continuously propagated within this step.

[0077] After completing the consistency comparison of the three-frame gesture categories and the consistency comparison of the three-frame pose descriptions, this step performs an average confidence threshold comparison and generates a gesture determination result. The average confidence score refers to the sum of the confidence scores within a group of the three frames. This step performs a validity screening on the confidence scores of the three frames in the dual-model alignment result structure. The validity screening includes checking whether the confidence score is missing, whether the confidence score corresponds one-to-one with the gesture category, and whether the confidence score is marked as unusable by an input anomaly flag. When unusable confidence scores exist, this step writes a missing confidence score flag and sets the average confidence score to the unusable flag, then proceeds to the determination result output branch. Understandably, when all three frame confidence scores are usable, this step performs an intra-group averaging operation on the three frame confidence scores to obtain the average confidence score, and compares the average confidence score with a preset confidence threshold. The preset confidence threshold is recorded in the confidence threshold configuration, which is an updatable configuration item maintained by a loop control configuration structure. This step combines the consistency comparison result with the average confidence comparison result. If the three-frame gesture category consistency comparison and the three-frame pose description consistency comparison are both marked as consistent, and the average confidence comparison meets the threshold condition, then the candidate gesture category is written into the gesture determination result and bound to the average confidence, frame number range, and frame timestamp range. If any condition is not met, then the gesture determination result is written to the abandonment determination mark and the triggering reason is recorded. The triggering reason includes at least one or more of the following: category conflict mark, pose inconsistency mark, and average confidence comparison not met mark. Further, this step generates a result record index for the gesture determination result. The result record index is bound to the dual model parameter version identifier, keyframe index structure reference index, and valid gesture image content structure reference index for cross-step tracing of the determination link. The gesture determination result is recorded as the output field name of this step and is read by the subsequent actuator interface or upper-layer business logic. It also serves as an input reference item when updating the loop control configuration.

[0078] Further, after generating the gesture determination result, this step performs a loop control configuration update process, constructs a loop control configuration structure, and writes it back to the upstream process. The loop control configuration structure refers to a set of configurations used to control sampling and determination in subsequent rounds. This set of configurations includes at least a fixed interval sampling configuration and a confidence threshold configuration. The fixed interval sampling configuration is used to adjust the fixed interval sampling and the extraction rhythm of three consecutive keyframes in S100, while the confidence threshold configuration is used to adjust the average confidence threshold comparison processing in this step. When updating the loop control configuration structure, this step reads the consistency comparison result, average confidence, input anomaly marker, and trigger reason from the gesture determination result, and generates the configuration for the next round according to preset update rules. These preset update rules include sampling interval adjustment rules, resampling trigger rules, and threshold preservation rules. These rules are stored in the loop control rule table and loaded during system initialization. Specifically, when a resampling entry or an input anomaly flag is triggered in this round, this step writes the fixed interval sampling configuration into the reset flag and records the reset reason. Subsequent steps in S100 reset the sampling starting point accordingly. When a category conflict flag or pose inconsistency flag appears in this round, this step writes the fixed interval sampling configuration into the adjustment flag and records the adjustment direction description, while maintaining the current version number of the confidence threshold configuration. When a missing average confidence flag appears in this round, this step writes the confidence threshold configuration into the rollback flag and records the rollback version number. The loop control configuration structure is recorded as an output field name at the end of this step and written back to S100 as the input source for the fixed interval sampling configuration. This allows S100 to re-enter S200 after generating the keyframe index structure in subsequent rounds and form a valid gesture image content structure, then enter S300 to construct the dual-model alignment result structure, and finally return to this step to complete the continuous operation loop.

[0079] The core of the gesture determination stage (corresponding to step S400) lies in consistency and confidence judgment. This module receives the results from dual-model parallel inference: the gesture category and corresponding confidence score output from a multimodal classification model (such as SigmaPipe2), and the coordinates of multiple joints and their derived pose descriptions output from a joint recognition model (such as MediaPipe Hands). The judgment logic is as follows: First, compare the consistency of the gesture category across three frames, requiring that the classification results of three consecutive frames are completely identical; second, compare the consistency of the pose description across three frames, requiring that the geometric features calculated based on the joint coordinates (such as specific fingertip distances and finger angles) change smoothly across the three frames, conforming to preset continuous action constraints; finally, calculate the average confidence score of the three frames and compare it with a preset confidence threshold (e.g., 90%). Only when all three conditions (consistent category, consistent pose, and average confidence score) are met simultaneously will the system finally determine and output a valid gesture; if any condition is not met, the system abandons this determination, and this result will also trigger an update of the loop control configuration to adjust subsequent video sampling or processing strategies. This fusion decision strategy significantly improves the accuracy and robustness of gesture recognition in complex scenarios through dual verification in both the spatiotemporal dimension (three consecutive frames) and the modal dimension (classification and pose).

[0080] Example 2: Figure 4 A structural block diagram of a gesture recognition system based on video stream analysis according to an embodiment of the present invention is shown. Figure 4 As shown, the structure may include:

[0081] The video stream acquisition and keyframe extraction module 01 is used to acquire the video acquisition device configuration and video stream frame images, perform frame timestamp writing, fixed-interval sampling, and extraction of three consecutive keyframes, and bind keyframe indices to generate a keyframe index structure. Specifically, the video stream acquisition and keyframe extraction module receives the video acquisition device configuration and establishes an acquisition channel, writes the video stream frame images into a frame buffer and writes a frame timestamp for each frame, and the frame timestamp and frame sequence number form an associated record. Under the fixed-interval sampling constraint, a sampling starting point is selected from the frame buffer according to the sampling interval, and three consecutive keyframes are extracted at the sampling starting point. The three keyframes are then associated with their corresponding frame sequences. The keyframe index binding record is formed by binding the frame number and the corresponding frame timestamp. When a frame is missing or the timestamp is abnormal in the acquisition channel, the abnormality mark and the corresponding frame number are written into the keyframe index binding record and the keyframe group in this round is invalidated. The keyframe index binding records are summarized to form a keyframe index structure. The keyframe index structure is provided as an input object to the hand area positioning and effective gesture image generation module. At the same time, it receives the loop control configuration structure written back by the consistency joint judgment and loop control module and updates the fixed interval sampling related configuration accordingly, so that the subsequent keyframe index structure generation process is consistent with the version association of the loop control configuration structure.

[0082] The hand region localization and effective gesture image generation module 02 is used to receive the keyframe index structure and perform YOLOv5s hand region localization processing, including hand region cropping and expansion, size normalization, and contrast enhancement operations, to obtain the effective gesture image content structure. Specifically, the hand region localization and effective gesture image generation module receives the keyframe index structure from the video stream acquisition and keyframe capture module, parses three keyframes and their bound frame numbers and timestamps, checks for abnormal markers in the keyframe index structure, and removes invalid keyframe groups. Hand region localization processing is performed on the three retained keyframes to obtain the hand region position description corresponding to each frame and associate it with the frame number. When the output of any frame's hand region localization processing is missing, the missing keyframe is... The invalid image content set is written into the dataset, retaining the invalidity reason identifier, corresponding frame number, and corresponding frame timestamp. For keyframes containing hand region location descriptions, the hand image region is cropped according to the hand region cropping and expansion rules, and boundary correction is performed. Subsequently, the cropped result is normalized in size and contrast enhancement is performed. The three normalized gesture images are bound to the corresponding frame number and corresponding frame timestamp in the same frame and written into the image quality mark to form a valid gesture image content structure. The valid gesture image content structure is provided as an input object to the dual-model training and parallel inference module, while the invalid image content set is provided as an access object to the dual-model training and parallel inference module, so that the training sample assembly process obtains consistent field association between valid sample entries and invalid sample entries.

[0083] The dual-model training and parallel inference module 03 is used to receive the effective gesture image content structure and access the invalid image content set and the keypoint annotation dataset, perform training sample assembly and dual-model training operations, and execute three-frame parallel inference and pose description generation processing to construct the dual-model alignment result structure. Specifically, the dual-model training and parallel inference module receives the effective gesture image content structure from the hand region localization and effective gesture image generation module, and accesses the invalid image content set and the keypoint annotation dataset. It performs consistency verification processing on the three types of inputs, checking the alignment relationship between frame sequence number and frame timestamp, the readable status of image references, the validity of the coordinate annotation boundaries of the twenty-one human hand keypoints in the keypoint annotation dataset, and the annotation consistency markers. Abnormal entries are written into the training data cleaning record and removed from the training sample assembly. In the training sample assembly stage, three normalized gesture images and image quality markers are extracted from the effective gesture image content structure to form effective sample entries, invalid reason identifiers and corresponding frame sequence numbers are extracted from the invalid image content set to form invalid sample entries, and twenty-one keypoints are extracted from the keypoint annotation dataset. The coordinates of key points on the human hand are labeled to form joint point sample entries. Sample balancing, training subset and validation subset partitioning, and sample versioning are then performed on these entries. During the dual-model training phase, parameter updates are performed and written to the model version record, ensuring consistency between the subsequent three-frame parallel inference calls and the model version record. In the three-frame parallel inference phase, three normalized gesture images are input into the inference pipeline in parallel, outputting three-frame gesture categories and three-frame confidence scores, along with joint point inference results in the same format as the joint point labeled dataset. Empty outputs and out-of-bounds entries are marked with inference anomaly flags. In the pose description generation phase, fingertip relative distance features, interfinal angle features, and horizontal displacement features are extracted from the joint point inference results and bound to the same frame. Then, the three-frame gesture categories, three-frame confidence scores, three-frame pose descriptions, and corresponding frame numbers and timestamps are aligned to the same frame and summarized within the group, forming a dual-model aligned result structure. This dual-model aligned result structure is provided as input to the consistency joint judgment and loop control module, forming a direct data source for subsequent consistency comparison and threshold comparison processing.

[0084] The consistency joint judgment and loop control module 04 is used to receive the dual-model alignment result structure, perform three-frame gesture category consistency comparison, three-frame pose description consistency comparison, and average confidence threshold comparison processing, generate gesture judgment results, construct a loop control configuration structure, and write the loop control configuration structure back to the video stream acquisition and keyframe capture module. Specifically, the consistency joint judgment and loop control module receives the dual-model alignment result structure from the dual-model training and parallel inference module, checks the continuity of frame sequence numbers and the consistency of frame timestamp order, and screens inference anomaly markers, writing anomaly entries into the resampling record; for dual-model alignment result structures that meet the verification conditions, it performs three-frame gesture category consistency comparison, outputs consistency markers, and associates candidate gesture categories with corresponding frame sequence number intervals; simultaneously, it performs three-frame pose description consistency comparison, calculates intra-group differences for fingertip relative distance features, finger-to-fingertip angle features, and horizontal displacement features, and outputs pose consistency markers; in the average confidence threshold comparison processing, it performs validity screening on the three-frame confidence and calculates the average confidence, and sets the average confidence... The degree and confidence threshold configuration are compared to form a threshold comparison result; based on the consistency label, posture consistency label and threshold comparison result, a joint judgment is made to generate a gesture judgment result and write it into the result record index, so that the gesture judgment result is traceably associated with the dual model alignment result structure, effective gesture image content structure and key frame index structure; in the process of constructing the loop control configuration structure, the fixed interval sampling related configuration and confidence threshold configuration are updated according to the record to be resampled and the joint judgment process record to form a loop control configuration structure and write it back to the video stream acquisition and key frame extraction module, so that the subsequent video stream frame image acquisition and three consecutive key frame extraction processes complete the configuration update according to the loop control configuration structure and enter the next round of closed-loop processing.

Claims

1. A gesture recognition method based on video stream analysis, characterized in that, include: Acquire video capture device configuration and video stream frame images, perform frame timestamp writing, fixed-interval sampling and extraction of three consecutive keyframes, keyframe index binding processing, and generate keyframe index structure; Based on the keyframe index structure, YOLOv5s hand region localization processing is performed, followed by hand region cropping and expansion, size normalization, and contrast enhancement processing to obtain an effective gesture image content structure. Based on the effective gesture image content structure, training sample assembly and dual-model training operations are performed, and three-frame parallel inference and pose description generation processing are executed to construct the dual-model aligned result structure. Based on the dual-model alignment result structure, the consistency comparison of three-frame gesture categories, the consistency comparison of three-frame pose descriptions, and the comparison of the average confidence threshold are performed to generate gesture judgment results and construct a loop control configuration structure. The process of performing three-frame parallel inference and pose description generation to construct the dual-model aligned result structure specifically includes: The three-frame parallel inference processing involves feeding three normalized gesture images into the multimodal classification model inference pipeline and the key point recognition model inference pipeline in parallel, and outputting the three-frame gesture category, three-frame confidence and three-frame key point coordinates. The posture description generation process includes extracting fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features from the three frames of joint coordinates, and performing noise reduction and smoothing processing on the joint coordinates. Calculate each feature according to the key point index relationship, and bind the feature results to the same frame; The three-frame gesture category, three-frame confidence, three-frame key point coordinates, three-frame pose description, and corresponding frame number and timestamp are aligned within the same frame and summarized within the group to generate a dual-model alignment result structure.

2. The method according to claim 1, characterized in that, The process of writing frame timestamps also includes: The frame timestamp writing process includes providing time stamps via a monotonically increasing system clock or acquisition hardware clock. When both hardware clock and system clock sources exist, a clock alignment table is used to complete time source mapping, converting the hardware timestamps into a unified system timestamp. Simultaneously, a video frame sequence is generated as an intermediate product. The video frame sequence contains a set of frame records enqueued in ascending order of frame number. The set of frame records includes frame content, frame number, frame timestamp, acquisition channel identifier, and anomaly flag. A circular buffer is used to maintain the most recent consecutive frame window.

3. The method according to claim 1, characterized in that, The fixed-interval sampling process also includes: Fixed-interval sampling processing involves selecting a sampling start point from a candidate frame window in the video frame sequence according to the sampling interval. The sampling interval is given by a fixed-interval sampling configuration, which includes a sampling interval value, a start offset, and an allowed number of frame skips, as well as a sampling start point reset condition. The sampling start point reset condition refers to triggering a reset operation when a switching of the acquisition channel identifier is detected, a clock back-back flag is detected during continuity checks, or an abnormal flag appears consecutively.

4. The method according to claim 1, characterized in that, The process of extracting three consecutive keyframes and binding keyframe indexes also includes: The three consecutive keyframe extraction process involves reading three consecutive frame records from the sampling start point, and the three frame records must have the same acquisition channel identifier and consecutive frame numbers. If a missing frame is encountered, the next frame is searched to fill the missing frame according to the allowed number of frame skips. Keyframe index binding is performed to generate a keyframe index structure, which includes references to the three keyframe images, corresponding frame numbers, corresponding frame timestamps, acquisition channel identifiers, anomaly marker summaries, and incomplete markers.

5. The method according to claim 1, characterized in that, The process of performing YOLOv5s hand region localization also includes: The hand region localization process uses the YOLOv5s object detection model to process three keyframe images frame by frame. The YOLOv5s object detection model includes a model weight file, a class mapping table, input size configuration, and inference threshold configuration. The inference threshold configuration includes a candidate box confidence threshold and an overlap suppression threshold. The hand region localization process includes feeding the three keyframe images frame by frame into the inference pipeline to perform pixel format alignment and scale adjustment, outputting a candidate bounding box set, and performing overlap suppression processing on the candidate bounding box set to form a valid bounding box set. When the valid bounding box set is empty, a non-detected flag is written and the current round of processing ends. When there are multiple valid bounding boxes, they are sorted by bounding box confidence and the main bounding box is selected based on the bounding box area constraint.

6. The method according to claim 1, characterized in that, The process of cropping and expanding the hand region, normalizing its size, and enhancing its contrast to obtain an effective gesture image content structure also includes: The hand region cropping and expansion operation involves expanding the width and height of the bounding box outwards from the main bounding box. The expansion ratio is defined as expanding the width and height of the bounding box by 10% each. After expansion, boundary cropping correction is performed. For frames with displacement or scale anomaly markers, cropping consistency correction is also performed, smoothly blending the coordinates of the expanded bounding box of that frame with the coordinates of the expanded bounding boxes of adjacent frames. Next, size normalization and contrast enhancement operations are performed. The size normalization operation involves scaling the cropped gesture image to a uniform spatial size of 384 pixels in both width and height, using a scaling strategy that maintains the aspect ratio. When the aspect ratio is inconsistent, edge padding is applied to the longer side. The contrast enhancement operation involves performing block histogram equalization on the size normalized image to obtain local contrast improvement. Finally, an effective gesture image content structure is generated. The effective gesture image content structure includes three normalized gesture images, three image quality markers, corresponding frame numbers, and corresponding frame timestamps. The image quality markers include sharpness description, brightness description, motion blur description, and anomaly marker summary.

7. The method according to claim 1, characterized in that, The process of comparing the consistency of gesture categories across three frames also includes: The three-frame gesture category consistency comparison process includes comparing the gesture categories of the three frames in the same group, comparing whether the categories of adjacent frames are the same in order of frame number and checking whether all three frame categories are the same, outputting a consistency flag, and when they are consistent, the common category is used as a candidate gesture category, and when they are inconsistent, a category conflict flag is written.

8. The method according to claim 1, characterized in that, The process of comparing the consistency of three-frame pose descriptions with the average confidence threshold also includes: The three-frame pose description consistency comparison process includes extracting fingertip relative distance features, finger-to-finger angle features, and horizontal displacement features from the three-frame pose descriptions, and performing intra-frame consistency checks and cross-frame consistency checks on each type of sub-feature set. The intra-frame consistency check includes verifying whether the joint index is complete, and the cross-frame consistency check includes judging whether the feature changes meet the constraints through a preset threshold set, and outputting a pose consistency flag. The average confidence threshold comparison process involves performing validity screening on the confidence of three frames and calculating the average confidence, then comparing the average confidence with a preset confidence threshold.

9. A gesture recognition system based on video stream analysis, applied to the method of any one of claims 1-8, characterized in that, include: The video stream acquisition and keyframe extraction module is used to acquire the video acquisition device configuration and video stream frame images, write frame timestamps, sample at fixed intervals and extract three consecutive keyframes, bind keyframe indexes and generate keyframe index structures. The hand region localization and effective gesture image generation module is used to receive the keyframe index structure and perform YOLOv5s hand region localization processing, perform hand region cropping and expansion, size normalization and contrast enhancement operations to obtain the effective gesture image content structure. The dual-model training and parallel inference module is used to receive the effective gesture image content structure and access the invalid image content set and key point annotation dataset, perform training sample assembly and dual-model training operations, and execute three-frame parallel inference and pose description generation processing to construct the dual-model aligned result structure. The consistency joint judgment and loop control module is used to receive the dual-model alignment result structure, perform three-frame gesture category consistency comparison, three-frame pose description consistency comparison and average confidence threshold comparison processing, generate gesture judgment results and construct loop control configuration structure, and write back the loop control configuration structure to the video stream acquisition and key frame capture module.

Images