Human-computer interaction control method and control system

Abstract

The present invention provides a human-computer interaction control method and a control system, and relates to the technical field of human-computer interaction. The present invention improves the robustness and accuracy of gesture recognition under different environmental conditions through multi-step image preprocessing such as light processing, angle correction, distance compensation and background separation, and can effectively improve the performance of gesture recognition in particular for situations of light changes and complex backgrounds. A dynamic weighted fusion algorithm is used to fuse image frames that have undergone multiple preprocessings, thereby ensuring the quality of the final integrated image frame, and can more accurately capture the features of gesture movements, thereby improving the accuracy of gesture recognition and making gesture recognition more fluent and natural. The method is not only applicable to the recognition of static gestures, but also supports the continuous recognition of dynamic gestures, and has strong adaptability and expansibility.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a human-computer interaction control method and a control system. Background Art

[0002] With the development of information technology, human-computer interaction methods are also constantly improving, from the initial traditional input devices such as keyboards and mice to a variety of new interaction methods such as touch screens, voice recognition, gesture recognition, etc. Gesture recognition, as an intuitive and natural human-computer interaction method, has received widespread attention. However, existing gesture recognition systems still face many challenges in practical applications, such as changes in ambient light, differences in gesture angles, and the distinction between hands and backgrounds. These problems may lead to a decrease in the accuracy of gesture recognition and a poor user experience. Traditional gesture recognition methods often rely on specific environmental conditions, such as stable light and fixed backgrounds, which limits their applicability in complex environments. In addition, for gesture recognition at different distances, existing technologies may not provide consistent recognition results.

[0003] Therefore, it is necessary to provide a human-computer interaction control method and a control system to solve the above technical problems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a human-computer interaction control method and a control system, which achieve the beneficial effect of being applicable in complex environments.

[0005] The present invention provides a human-computer interaction control method, which comprises the following steps:

[0006] S1: obtaining a gesture action video stream containing a complete gesture action, and reading the gesture action video stream frame by frame to obtain a plurality of gesture action image frames;

[0007] S2: traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain a light compensation image frame, an angle correction image frame, a distance compensation image frame and a background separation image frame corresponding to each gesture action image frame;

[0008] S3: traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame;

[0009] S4: traversing and dynamically weighting and fusing each light-compensated image frame with the corresponding first aligned image frame, second aligned image frame, and third aligned image frame to obtain a plurality of integrated image frames;

[0010] S5: extracting gesture action features from a plurality of comprehensive image frames and combining them into a gesture action feature sequence, and inputting the gesture action feature sequence into a pre-trained gesture action recognition model for recognition, and generating corresponding control instructions according to the recognition result.

[0011] Preferably, in step S2, the light compensation image frame is obtained by calculating the average illumination intensity of each gesture action image frame, and dynamically adjusting the brightness and contrast of the gesture action image frame based on the average illumination intensity, and the adjustment formula is:

[0012] I i =α·I t +β

[0013] α=1+k1(L t -L ref )

[0014] β=k2(L t -L ref )

[0015] Among them, α is the gain factor, β is the offset factor, I i is the light compensation image frame, I t is the gesture action image frame, L t is the average illumination intensity of the gesture action image frame, L ref is the reference light intensity, k1 and k2 are adjustment coefficients.

[0016] Preferably, in step S2, each light-compensated image frame is traversed and image quality assessment is performed to obtain an image quality score for each light-compensated image frame.

[0017] Preferably, the angle-corrected image frame is obtained by estimating the rotation angle of the gesture action image frame using a feature point detection and matching algorithm and performing rotation correction.

[0018] Preferably, the distance-compensated image frame is obtained by acquiring a depth image using an image acquisition device, and scaling the gesture action image frame based on an image scaling ratio calculated based on a depth feature in the depth image.

[0019] Preferably, the background separated image frame is obtained by separating the hand gesture from the background of the hand gesture image frame by using a background subtraction method in combination with depth features.

[0020] Preferably, step S3 comprises the following steps:

[0021] S301: traverse and use a feature point detection algorithm to respectively detect feature points of each light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame;

[0022] S302: traverse and use a feature point matching algorithm to match corresponding feature points between each light compensation image frame and the corresponding angle correction image frame, distance compensation image frame and background separation image frame, respectively, to obtain a first feature point pair set, a second feature point pair set, and a third feature point pair set;

[0023] S303: Based on the first feature point pair set, the second feature point pair set, and the third feature point pair set, respectively calculating the affine transformation matrix between the light compensation image frame and the corresponding angle correction image frame, distance compensation image frame, and background separation image frame;

[0024] S304: Based on the corresponding affine transformation matrix, align the light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame into the same coordinate system to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame.

[0025] Preferably, in step S4, the formula for dynamically weighted fusion of the light-compensated image frame with the corresponding first aligned image frame, second aligned image frame and third aligned image frame is:

[0026] I final =ω0I i +ω1I r +ω2I s +ω3I b

[0027] Among them, I final is the comprehensive image frame, I i is the light compensation image frame, I r Align the image frames for angle correction, I s Align image frames for distance compensation, I b The aligned image frames are for background separation, and ω0, ω1, ω2, and ω3 are the fusion weights of the light compensation image frames, the first aligned image frames, the second aligned image frames, and the third aligned image frames, respectively.

[0028] Preferably, the step of determining each fusion weight includes:

[0029] Obtaining an initial fusion weight of the light-compensated image frame based on an image quality score of the light-compensated image frame and a preset mapping table;

[0030] Respectively calculating the inlier ratio, average matching distance and number of feature point pairs of the first feature point pair set, the second feature point pair set and the third feature point pair set corresponding to the light-compensated image frame, and calculating the matching degree scores corresponding to the first feature point pair set, the second feature point pair set and the third feature point pair set based on the obtained inlier ratio, average matching distance and number of feature point pairs;

[0031] The obtained matching degree scores are normalized to obtain a first initial fusion weight, a second initial fusion weight, and a third initial fusion weight respectively;

[0032] The initial fusion weight of the light compensation image frame, the first initial fusion weight, the second initial fusion weight and the third initial fusion weight are normalized to obtain the light compensation image frame fusion weight, the first alignment image frame fusion weight, the second alignment image frame fusion weight and the third alignment image frame fusion weight.

[0033] The present invention also provides a human-computer interaction control system, which is applied to a human-computer interaction control method. The human-computer interaction control system comprises:

[0034] Video stream processing module: used to obtain a gesture action video stream containing a complete gesture action, and read the gesture action video stream frame by frame to obtain multiple gesture action image frames;

[0035] Image preprocessing module: used to traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain the light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame corresponding to each gesture action image frame;

[0036] Image alignment module: used to traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame;

[0037] Image fusion module: used for traversing and dynamically weighting fusion of each light compensation image frame with the corresponding first alignment image frame, second alignment image frame and third alignment image frame to obtain multiple comprehensive image frames;

[0038] Gesture recognition control module: used to extract gesture action features from multiple comprehensive image frames and combine them into a gesture action feature sequence, and input the gesture action feature sequence into a pre-trained gesture action recognition model for recognition, and generate corresponding control instructions based on the recognition results.

[0039] Compared with the related art, the human-computer interaction control method and control system provided by the present invention have the following beneficial effects:

[0040] The present invention improves the robustness and accuracy of gesture recognition under different environmental conditions through multi-step image preprocessing such as light processing, angle correction, distance compensation and background separation. In particular, it can effectively improve the performance of gesture recognition under conditions of light changes and complex backgrounds. A dynamic weighted fusion algorithm is used to fuse image frames that have undergone multiple preprocessing processes to ensure the quality of the final integrated image frame. The features of gesture movements can be captured more accurately, thereby improving the accuracy of gesture recognition and making gesture recognition smoother and more natural. This method is not only suitable for the recognition of static gestures, but also supports the continuous recognition of dynamic gestures, and has strong adaptability and expansibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a human-computer interaction control method of the present invention;

[0042] Figure 2 The present invention is a schematic diagram of the module structure of a human-computer interaction control system. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0044] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0045] Embodiment 1

[0046] A human-computer interaction control method, in the specific implementation process, such as Figure 1 As shown, it shows a schematic diagram of a human-computer interaction control method flow chart, including:

[0047] Step S1: obtaining a gesture action video stream containing a complete gesture action, and reading the gesture action video stream frame by frame to obtain a plurality of gesture action image frames.

[0048] During the specific implementation process, first, a video acquisition device is used to capture the user's gesture action video stream in real time, then a video decoder is used to decode the received gesture action video stream into an original image frame sequence, and finally, a video processing library is used to read the decoded gesture action image frames frame by frame to obtain continuous image frames containing gesture action information.

[0049] Step S2: traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain a light compensation image frame, an angle correction image frame, a distance compensation image frame and a background separation image frame corresponding to each gesture action image frame.

[0050] During the specific implementation process, each gesture action image frame containing complete gesture action information is traversed, and light processing is performed on each gesture action image frame to obtain a light compensation image frame with better quality; angle correction is performed on each gesture action image frame to obtain an angle correction image frame; distance compensation is performed on each gesture action image frame, and the distance compensation image frame is obtained by scaling the gesture action image frame; background separation is performed on each gesture action image frame, and the separated gesture action image is used as the background separation image frame.

[0051] Specifically, in step S2, the light compensation image frame is obtained by calculating the average light intensity of each gesture action image frame, and dynamically adjusting the brightness and contrast of the gesture action image frame based on the average light intensity. The adjustment formula is:

[0052] I i =α·I t +β

[0053] α=1+k1(L t -L ref )

[0054] β=k2(L t -L ref )

[0055] Among them, α is the gain factor, β is the offset factor, I i is the light compensation image frame, I t is the gesture action image frame, L t is the average illumination intensity of the gesture action image frame, L ref is the reference light intensity, k1 and k2 are adjustment coefficients.

[0056] In the specific implementation process, under different lighting conditions, the brightness and contrast of the image may vary greatly. By dynamically adjusting the gain and offset, the image can maintain a consistent visual effect under different lighting conditions. Gain and offset are two important parameters used to adjust the brightness and contrast of the image. They are used to enlarge or reduce the pixel value of the image and increase or decrease the brightness of the image respectively. The gain factor α is a multiplication factor used to adjust the brightness and contrast of the image. When adjusting the brightness, when the gain factor α is greater than 1, the image will become brighter, and when it is less than 1, the image will become darker; when adjusting the contrast, the larger the gain factor α, the higher the contrast of the image; the smaller the gain factor α, the lower the contrast of the image. k1 is an adjustment coefficient used to control the change range of the gain. The offset factor β is an additive term used to adjust the overall brightness of the image. When the offset factor β is greater than 0, the image will become brighter; when it is less than 0, the image will become darker. k2 is an adjustment coefficient used to control the change range of the offset. L ref The reference light intensity may be, for example, but not limited to, using a light sensor to measure the light intensity in a typical environment as the reference light intensity, L t is the average illumination intensity of the gesture action image frame, which is obtained by calculating the grayscale value of each pixel in the gesture action image frame and finding the average grayscale value of all pixels to obtain the average illumination intensity of the current gesture action image frame.

[0057] Specifically, in step S2, each light-compensated image frame is traversed and image quality evaluation is performed to obtain an image quality score for each light-compensated image frame.

[0058] During the specific implementation process, exemplarily, the BRISQUE algorithm is used to calculate the image quality score of each light-compensated image frame, and the image quality score of each light-compensated image frame is stored for use in subsequent steps.

[0059] Specifically, the angle-corrected image frame is obtained by estimating the rotation angle of the gesture action image frame using a feature point detection and matching algorithm and performing rotation correction.

[0060] During the specific implementation process, the rotation angle of the gesture action image frame is estimated by using feature point detection and matching algorithm, and the gesture action image frame is rotationally corrected using a rotation transformation matrix according to the estimated rotation angle, so that the gesture action remains in the horizontal or vertical direction.

[0061] Specifically, the distance-compensated image frame is obtained by acquiring a depth image using an image acquisition device, and scaling the gesture action image frame based on an image scaling ratio calculated based on a depth feature in the depth image.

[0062] During the specific implementation process, the depth image corresponding to the gesture action image frame is obtained, the scaling ratio of the gesture action image frame is calculated according to the depth features in the depth image, and the gesture action image frame is scaled according to the calculated scaling ratio to obtain a distance compensated image frame. Exemplarily, the acquisition of the depth image includes but is not limited to a depth camera, a binocular camera, a monocular depth estimation, and structured light.

[0063] Specifically, the background separation image frame is obtained by separating the gesture action from the background of the gesture action image frame through a background subtraction method combined with a depth feature.

[0064] During the specific implementation process, first, a background modeling method is used, exemplarily including but not limited to Gaussian mixture model GMM and mean shift Mean Shift, to establish a background model, and then the background model is used to subtract the background from each gesture image frame to preliminarily separate the gesture area. Secondly, the gesture area is further refined in combination with deep features to remove background noise. Finally, the separated gesture area is post-processed, exemplarily including but not limited to morphological operations and filtering to improve the separation effect.

[0065] Step S3: traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame.

[0066] Specifically, step S3 includes the following steps:

[0067] S301: traverse and use a feature point detection algorithm to respectively detect feature points of each light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame;

[0068] S302: traverse and use a feature point matching algorithm to match corresponding feature points between each light compensation image frame and the corresponding angle correction image frame, distance compensation image frame and background separation image frame, respectively, to obtain a first feature point pair set, a second feature point pair set, and a third feature point pair set;

[0069] S303: Based on the first feature point pair set, the second feature point pair set, and the third feature point pair set, respectively calculating the affine transformation matrix between the light compensation image frame and the corresponding angle correction image frame, distance compensation image frame, and background separation image frame;

[0070] S304: Based on the corresponding affine transformation matrix, align the light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame into the same coordinate system to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame.

[0071] In the specific implementation process, exemplarily, the SIFT algorithm is first used to detect the feature points of each light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame respectively, and then the BFMatcher algorithm is used to match the corresponding feature points between each light compensation image frame and the corresponding angle correction image frame, distance compensation image frame and background separation image frame respectively, and the Rockwell ratio test is applied to screen out reliable matching point pairs, that is, the corresponding first feature point pair set, second feature point pair set, and third feature point pair set, and then the RANSAC algorithm is used to calculate the affine transformation matrix from the matching point pairs, and finally, the calculated affine transformation matrix is ​​used to align the angle correction image frame, the distance compensation image frame and the background separation image frame to the coordinate system of the light compensation image frame to obtain the first aligned image frame, the second aligned image frame and the third aligned image frame.

[0072] Step S4: traverse and dynamically weighted fuse each light-compensated image frame with the corresponding first aligned image frame, second aligned image frame and third aligned image frame to obtain a plurality of integrated image frames.

[0073] Specifically, in step S4, the formula for dynamically weighted fusion of the light-compensated image frame with the corresponding first aligned image frame, second aligned image frame and third aligned image frame is:

[0074] I final =ω0I i +ω1I r +ω2I s +ω3I b

[0075] Among them, I final is the comprehensive image frame, I i is the light compensation image frame, I r Align the image frames for angle correction, I s Align image frames for distance compensation, I b The aligned image frames are for background separation, and ω0, ω1, ω2, and ω3 are the fusion weights of the light compensation image frames, the first aligned image frames, the second aligned image frames, and the third aligned image frames, respectively.

[0076] During the specific implementation process, each light-compensated image frame needs to be dynamically weighted-fused with the corresponding first aligned image frame, second aligned image frame and third aligned image frame to obtain multiple composite image frames, thereby improving the image quality of the recognized gesture action image frame and improving the recognition accuracy.

[0077] Specifically, the steps of determining each fusion weight include:

[0078] Obtaining an initial fusion weight of the light-compensated image frame based on an image quality score of the light-compensated image frame and a preset mapping table;

[0079] Respectively calculating the inlier ratio, average matching distance and number of feature point pairs of the first feature point pair set, the second feature point pair set and the third feature point pair set corresponding to the light-compensated image frame, and calculating the matching degree scores corresponding to the first feature point pair set, the second feature point pair set and the third feature point pair set based on the obtained inlier ratio, average matching distance and number of feature point pairs;

[0080] The obtained matching degree scores are normalized to obtain a first initial fusion weight, a second initial fusion weight, and a third initial fusion weight respectively;

[0081] The initial fusion weight of the light compensation image frame, the first initial fusion weight, the second initial fusion weight and the third initial fusion weight are normalized to obtain the light compensation image frame fusion weight, the first alignment image frame fusion weight, the second alignment image frame fusion weight and the third alignment image frame fusion weight.

[0082] During the specific implementation process, first, it is necessary to determine the initial fusion weight of the light-compensated image frame, and obtain the initial fusion weight of the light-compensated image frame through the image quality score of the light-compensated image frame and the preset mapping table. The preset and mapping table are mapping tables about the image quality score of the light-compensated image frame and the specific weight value, which are determined and adjusted by experiments in the actual environment. The higher the image quality score of the light-compensated image frame, the higher the initial fusion weight of the light-compensated image frame. Then, the matching degree scores corresponding to the first feature point pair set, the second feature point pair set, and the third feature point pair set corresponding to the light-compensated image frame are calculated respectively. The matching degree scores are based on the inlier ratio, the average matching distance and the number of feature point pairs of the first feature point pair set, the second feature point pair set, and the third feature point pair set corresponding to the light-compensated image frame. Table The matching degree of the feature point pair is shown in Figure 2. The higher the matching degree, the higher the matching degree score, and the higher the fusion weight will be in the image fusion process. Secondly, the obtained matching degree score is normalized to obtain the first initial fusion weight, the second initial fusion weight and the third initial fusion weight respectively. Finally, the initial fusion weight of the light compensated image frame, the first initial fusion weight, the second initial fusion weight and the third initial fusion weight are normalized to obtain the light compensated image frame fusion weight, the first aligned image frame fusion weight, the second aligned image frame fusion weight and the third aligned image frame fusion weight, that is, the final image frame fusion weight. The dynamic acquisition step of each fusion weight fully considers the image quality and the feature matching degree of the image, and improves the accuracy and robustness of image fusion.

[0083] The working principle of a human-computer interaction control method provided by the present invention is as follows:

[0084] By acquiring a video stream containing complete gestures and processing it frame by frame, each gesture image frame is subjected to light processing, angle correction, distance compensation, and background separation to generate multiple pre-processed image frames. Then, these pre-processed image frames are fused into comprehensive image frames through spatial alignment and dynamic weighted fusion algorithms. Finally, gesture features are extracted and input into a pre-trained gesture recognition model to generate corresponding control instructions. This method significantly improves the robustness and accuracy of gesture recognition through multi-step image preprocessing and fusion, and is suitable for gesture recognition applications in a variety of complex environments.

[0085] Embodiment 2

[0086] A human-computer interaction control system is applied to a human-computer interaction control method. In the specific implementation process, Figure 2 As shown, it shows a schematic diagram of the module structure of a human-computer interaction control system, and the human-computer interaction control system includes:

[0087] The video stream processing module 100 is used to obtain a gesture action video stream containing a complete gesture action, and read the gesture action video stream frame by frame to obtain a plurality of gesture action image frames;

[0088] Image preprocessing module 200: used to traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain a light compensation image frame, an angle correction image frame, a distance compensation image frame and a background separation image frame corresponding to each gesture action image frame;

[0089] Image alignment module 300: used to traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame;

[0090] Image fusion module 400: used to traverse and dynamically weighted fuse each light-compensated image frame with the corresponding first aligned image frame, second aligned image frame and third aligned image frame to obtain multiple integrated image frames;

[0091] Gesture recognition control module 500: used to extract gesture action features from multiple comprehensive image frames and combine them into a gesture action feature sequence, and input the gesture action feature sequence into a pre-trained gesture action recognition model for recognition, and generate corresponding control instructions according to the recognition results.

[0092] The working principle of a human-computer interaction control system provided by the present invention is as follows:

[0093] The video stream processing module 100 acquires and reads the gesture video stream frame by frame, the image preprocessing module 200 performs light processing, angle correction, distance compensation and background separation on each gesture image frame to generate multiple preprocessed image frames, the image alignment module 300 performs spatial alignment on these preprocessed image frames, the image fusion module 400 generates a comprehensive image frame through a dynamic weighted fusion algorithm, and finally, the gesture recognition control module 500 extracts the gesture features in the comprehensive image frame, inputs the pre-trained gesture recognition model for recognition, and generates corresponding control instructions. The system significantly improves the accuracy and robustness of gesture recognition through multi-step image processing and fusion, and is suitable for gesture recognition applications in a variety of complex environments.

[0094] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A human-computer interaction control method, characterized in that: The human-computer interaction control method comprises the following steps: S1: obtaining a gesture action video stream containing a complete gesture action, and reading the gesture action video stream frame by frame to obtain a plurality of gesture action image frames; S2: traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain a light compensation image frame, an angle correction image frame, a distance compensation image frame and a background separation image frame corresponding to each gesture action image frame; S3: traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame; S4: traversing and dynamically weighting and fusing each light-compensated image frame with the corresponding first aligned image frame, second aligned image frame, and third aligned image frame to obtain a plurality of integrated image frames; S5: extracting gesture action features from a plurality of comprehensive image frames and combining them into a gesture action feature sequence, and inputting the gesture action feature sequence into a pre-trained gesture action recognition model for recognition, and generating corresponding control instructions according to the recognition result; In step S4, the formula for dynamically weighted fusion of the light-compensated image frame with the corresponding first aligned image frame, second aligned image frame and third aligned image frame is: I final =ω0I i +ω1I r +ω2I s +ω3I b Among them, I final is the comprehensive image frame, I i is the light compensation image frame, I r Align the image frames for angle correction, I s Align image frames for distance compensation, I b Align the image frames for background separation, ω0, ω1, ω2, and ω3 are the fusion weights of the light compensation image frame, the first alignment image frame, the second alignment image frame, and the third alignment image frame, respectively; The steps for determining each fusion weight include: Obtaining an initial fusion weight of the light-compensated image frame based on an image quality score of the light-compensated image frame and a preset mapping table; Respectively calculating the inlier ratio, average matching distance and number of feature point pairs of the first feature point pair set, the second feature point pair set and the third feature point pair set corresponding to the light-compensated image frame, and calculating the matching degree scores corresponding to the first feature point pair set, the second feature point pair set and the third feature point pair set based on the obtained inlier ratio, average matching distance and number of feature point pairs; The obtained matching degree scores are normalized to obtain a first initial fusion weight, a second initial fusion weight, and a third initial fusion weight respectively; The initial fusion weight of the light compensation image frame, the first initial fusion weight, the second initial fusion weight and the third initial fusion weight are normalized to obtain the light compensation image frame fusion weight, the first alignment image frame fusion weight, the second alignment image frame fusion weight and the third alignment image frame fusion weight.

2. A human-computer interaction control method according to claim 1, characterized in that: In step S2, the light compensation image frame is obtained by calculating the average illumination intensity of each gesture action image frame, and dynamically adjusting the brightness and contrast of the gesture action image frame based on the average illumination intensity. The adjustment formula is: I i =α·I t +b α=1+k1(L t -L ref ) β=k2(L t -L ref ) Among them, α is the gain factor, β is the offset factor, I i is the light compensation image frame, I t is the gesture action image frame, L t is the average illumination intensity of the gesture action image frame, L ref is the reference light intensity, k1 and k2 are adjustment coefficients.

3. A human-computer interaction control method according to claim 1, characterized in that: In step S2, each light-compensated image frame is traversed and image quality assessment is performed to obtain an image quality score for each light-compensated image frame.

4. A human-computer interaction control method according to claim 3, characterized in that: The angle-corrected image frame is obtained by estimating the rotation angle of the gesture action image frame using a feature point detection and matching algorithm and performing rotation correction.

5. A human-computer interaction control method according to claim 4, characterized in that: The distance compensation image frame is obtained by acquiring a depth image using an image acquisition device, and scaling the gesture action image frame based on an image scaling ratio calculated based on a depth feature in the depth image.

6. A human-computer interaction control method according to claim 5, characterized in that: The background separation image frame is obtained by separating the hand gesture from the background of the hand gesture image frame by using a background subtraction method in combination with a depth feature.

7. A human-computer interaction control method according to claim 6, characterized in that: Step S3 includes the following steps: S301: traverse and use a feature point detection algorithm to respectively detect feature points of each light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame; S302: traverse and use a feature point matching algorithm to match corresponding feature points between each light compensation image frame and the corresponding angle correction image frame, distance compensation image frame and background separation image frame, respectively, to obtain a first feature point pair set, a second feature point pair set, and a third feature point pair set; S303: Based on the first feature point pair set, the second feature point pair set, and the third feature point pair set, respectively calculating the affine transformation matrix between the light compensation image frame and the corresponding angle correction image frame, distance compensation image frame, and background separation image frame; S304: Based on the corresponding affine transformation matrix, align the light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame into the same coordinate system to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame.

8. A human-computer interaction control system, characterized in that: Applied to a human-computer interaction control method as claimed in any one of claims 1 to 7, the human-computer interaction control system comprises: Video stream processing module: used to obtain a gesture action video stream containing a complete gesture action, and read the gesture action video stream frame by frame to obtain multiple gesture action image frames; Image preprocessing module: used to traverse and perform light processing, angle correction, distance compensation and background separation operations on each gesture action image frame, and obtain the light compensation image frame, angle correction image frame, distance compensation image frame and background separation image frame corresponding to each gesture action image frame; Image alignment module: used to traverse and spatially align each obtained light compensation image frame with the corresponding angle correction image frame, distance compensation image frame and background separation image frame to obtain a first aligned image frame, a second aligned image frame and a third aligned image frame; Image fusion module: used for traversing and dynamically weighting fusion of each light compensation image frame with the corresponding first alignment image frame, second alignment image frame and third alignment image frame to obtain multiple comprehensive image frames; Gesture recognition control module: used to extract gesture action features from multiple comprehensive image frames and combine them into a gesture action feature sequence, and input the gesture action feature sequence into a pre-trained gesture action recognition model for recognition, and generate corresponding control instructions based on the recognition results.

Images