A gesture recognition system and method based on video coding

Abstract

The present invention discloses a gesture recognition system and method based on video coding, which relates to the field of Internet and audio and video streaming media development. The system collects gesture video data through a video acquisition module, then uses a video processing and analysis module to calculate a video coding determination coefficient, and analyzes whether the video meets the coding requirements by setting a threshold comparison method. Then, the encoding and decoding processes are performed on the video that meets the requirements through the encoding module and the decoding module to obtain the original video data. Then, the video extraction and analysis module extracts and calculates the original video data to obtain the internal features of the video, and performs quality calculation to obtain the video frame quality coefficient. The video frame quality is rated by performing a difference comparison method by setting a threshold, and key frames are screened. Then, the gesture recognition module processes the key frames and comprehensively calculates the hand position to obtain the hand position comprehensive coefficient. Then, the gesture recognition model is used to perform recognition to obtain the gesture recognition result.

Description

Technical Field

[0001] The present invention relates to the field of Internet and audio and video streaming media development, and in particular to a gesture recognition system and method based on video coding. Background Art

[0002] Dynamic gestures, as an important form of human-computer interaction, have garnered increasing attention in recent years. By capturing and analyzing specific hand movements using devices, these movements are converted into a series of control commands that trigger responses from smart devices, making human-computer interaction more natural and user-friendly. In recent years, dynamic gesture recognition has seen increasing application in areas such as somatosensory gaming, sign language recognition, assisted driving, medical devices, and smart home appliance control.

[0003] Currently, many projects have increasingly higher requirements for video integration, and the video resources that need to be integrated are becoming more and more abundant. Many gesture video streams are accessed using the H.265 encoding format. WebRTC itself only supports VP8, VP9, ​​H.264, and AV1 video encoding formats, and does not support H.265 encoding. As a result, the video cannot be retrieved during gesture recognition, which affects the recognition of video gestures. In addition, there are certain problems with the recognition accuracy of gesture recognition methods using videos. Summary of the Invention

[0004] To address the deficiencies mentioned in the above background technology, the present invention aims to provide a gesture recognition system and method based on video coding, which can perform gesture recognition after encoding and decoding various videos and can improve the gesture recognition accuracy.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a gesture recognition system based on video coding, comprising:

[0006] Video acquisition module: used to collect gesture video data and send the collected gesture video data to the video processing and analysis module for processing;

[0007] Video processing and analysis module: used to extract video features from gesture video data to obtain video feature-related data, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data; use the video feature-related data to perform coding determination calculation to obtain a video coding determination coefficient; set a video coding determination threshold; compare the video coding determination coefficient with the video coding determination threshold; and determine whether the video meets the coding requirements based on the comparison result. If not, the corresponding video is deleted; if it meets, a coding signal is sent to the coding module;

[0008] Encoding module: used to obtain the video stream corresponding to the gesture video data in the video acquisition module, then encode it to obtain the encoded video stream, and send the encoded video stream to the decoding module;

[0009] Decoding module: used to decode the encoded video stream to obtain the original video data;

[0010] Video extraction and analysis module: used to perform video interception on the original video data to obtain multiple video frame data, perform feature extraction on each video frame in the multiple video frame data after the video interception, obtain video internal features, wherein the video internal features include position feature data, motion feature data and shape feature data, perform quality calculation using the marked video internal features to obtain a video frame quality coefficient, set a video frame quality coefficient threshold, use the video frame quality coefficient and the video frame quality coefficient threshold to make a difference to obtain a difference, set a difference threshold, perform a ratio determination between the difference and the difference threshold, and rate the video frame quality according to the determination result, wherein the rated video levels include: low-level video frames, medium-level video frames and high-level video frames;

[0011] Count the number of low-level video frames. If the number of low-level video frames is less than one-third of the total number of video frames, mark the intermediate and high-level video frames as key frames and send the key frames to the gesture recognition module. If the number of low-level video frames reaches or exceeds one-third of the total number of video frames, delete the extracted video frames and recapture the video.

[0012] Gesture recognition module: used to establish an XY plane rectangular coordinate system, use the coordinate acquisition unit of the gesture recognition module to obtain the gesture key points in each key frame in the plane rectangular coordinate system, obtain the coordinates of the gesture key points, calculate the finger spacing based on the gesture key point coordinates, and extract the finger direction information. Use the finger spacing and finger direction information to perform a comprehensive calculation of the hand position to obtain the hand position comprehensive coefficient. The hand position comprehensive coefficient is input into a pre-established gesture recognition model to output the gesture recognition result;

[0013] Server: Used to integrate and obtain standard hand-related data required for training pre-established gesture recognition models.

[0014] In conjunction with the first aspect, in certain implementations of the first aspect, the system further includes: a calculation process of the video processing and analysis module:

[0015] Mark the video feature related data, where the resolution feature data is marked as Bi, the frame rate feature data is marked as Zi, and the segmentation feature data is marked as Gi;

[0016] The coding decision calculation is performed using the marked video feature related data. The coding decision calculation formula is as follows:

[0017]

[0018] Where Pmi is the video coding determination coefficient, B0 is the standard resolution feature coefficient, Z0 is the standard frame rate feature coefficient, G0 is the standard segmentation feature coefficient, k1 is the resolution correlation coefficient, k2 is the frame rate correlation coefficient, k3 is the segmentation correlation coefficient, and R is the preset correlation coefficient.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, the system further includes: an analysis process of the video processing and analysis module:

[0020] Set the video coding determination threshold Pm0, and compare the video coding determination coefficient Pmi with the video coding determination threshold Pm0 to determine:

[0021] If Pmi≥Pm0, it is determined that the video meets the encoding requirements and the encoding signal is sent to the encoding module;

[0022] If Pmi<Pm0, it is determined that the video does not meet the encoding requirements at this time, and the video corresponding to the video encoding determination coefficient is deleted.

[0023] In conjunction with the first aspect, in certain implementations of the first aspect, the system further includes: a calculation process of the video extraction and analysis module:

[0024] Mark the internal features of the video, where the position feature data is marked as Wj, the motion feature data is marked as Yj, and the shape feature data is marked as Xj, where j is the number of internal features of the video, and j=1, 2, 3, ..., m, and m is the total number of internal features of the video;

[0025] The quality of the video frame is calculated using the marked internal features to obtain the video frame quality coefficient. Specifically, the calculation process of the video frame quality coefficient is as follows:

[0026]

[0027] Where Szj is the video frame quality coefficient, T1 is the position feature influence coefficient, T2 is the motion feature influence coefficient, T3 is the shape feature influence coefficient, X0 is the preset annotation shape feature coefficient, and α is the preset shape ratio coefficient.

[0028] In conjunction with the first aspect, in certain implementations of the first aspect, the system further includes: an analysis process of the video extraction and analysis module:

[0029] Set the video frame quality coefficient threshold Sz0, use the video frame quality coefficient Szj to make a difference with the video frame quality coefficient threshold Sz0, obtain the difference Cj, set the difference threshold C0, and perform a proportional determination between the difference Cj and the difference threshold C0:

[0030] like Determining the video frame quality as low-level video frame;

[0031] like The video frame quality is determined to be medium-level;

[0032] like Determine the video frame quality as high-quality video frame;

[0033] Count the number of low-level video frames, denoted as q, if At this point, the extracted video frames meet the recognition standard number, the intermediate video frames and the advanced video frames are marked as key frames, and the key frames are sent to the gesture recognition module for gesture recognition;

[0034] like If the extracted video frames do not meet the recognition standard number, the extracted video frames will be deleted and a re-collection signal will be sent to the video collection module. until.

[0035] In combination with the first aspect, in certain implementations of the first aspect, the system further includes: the gesture key points in the gesture recognition module include a first key point, a second key point, a third key point, a fourth key point, and a fifth key point;

[0036] Among them, the first key point is marked as A1(X 1u , Y 1u ), the second key point is marked as A2(X 2u , Y 2u ), the third key point is marked as A3(X 3u , Y 3u ), the fourth key point is marked as A4(X 4u , Y 4u ); the fifth key point is marked as A5(X 5u , Y 5u ), where u is the number of each key frame, and u = 1, 2, 3, ..., V; V is the total number of key frames.

[0037] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the gesture recognition module calculates the finger distance using the coordinates of the first key point, the second key point, the third key point, the fourth key point, and the fifth key point, wherein the distance between one and two fingers is marked as a 12u , the distance between two and three fingers is marked as a 23u , the distance between three and four fingers is marked as a 34u , the distance between four and five fingers is marked as a 45u ;

[0038] Extract the direction information of each finger;

[0039] The finger distance and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the comprehensive hand position coefficient:

[0040]

[0041] Where sub is the comprehensive coefficient of hand position, l1, l2, l3, and l4 are all preset direction correlation coefficients, and β is the preset position proportional coefficient.

[0042] In conjunction with the first aspect, in certain implementations of the first aspect, the system further includes: the gesture recognition model pre-established by the gesture recognition module is trained based on an artificial intelligence model;

[0043] Integrating and acquiring standard hand-related data through a server, wherein the standard hand-related data includes standard finger spacing data, standard finger position coordinate data, and standard finger direction data;

[0044] The artificial intelligence model is trained using standard hand-related data to obtain and store the gesture recognition model.

[0045] In combination with the first aspect, in some implementations of the first aspect, the system further includes: the decoding module is capable of directly decoding the H.265 format video stream encoded by the encoding module

[0046] In conjunction with the second aspect, in certain implementations of the second aspect, the method further includes: a gesture recognition method based on video coding, the method comprising the following steps:

[0047] Acquiring gesture video data, performing video feature extraction on the gesture video data to obtain video feature-related data, and performing coding determination calculation using the video feature-related data to obtain a video coding determination coefficient, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data;

[0048] Set the video coding determination threshold, compare the video coding determination coefficient with the video coding determination threshold, and determine whether the video meets the coding requirements based on the comparison result. If not, delete the corresponding video. If it meets the requirements, perform the encoding and decoding process on the video, and obtain the original video data after decoding;

[0049] Performing video interception on original video data to obtain multiple video frame data, performing feature extraction on each video frame in the multiple video frame data after the video interception to obtain video internal features, and performing quality calculation using the video internal features to obtain a video frame quality coefficient, wherein the video internal features include position feature data, motion feature data, and shape feature data;

[0050] Setting a video frame quality coefficient threshold, performing a difference calculation between the video frame quality coefficient and the video frame quality coefficient threshold to obtain a difference, setting a difference threshold, performing a proportional determination between the difference and the difference threshold, and rating the video frame quality according to the determination result, wherein the rated video levels include: low-level video frame, medium-level video frame, and high-level video frame;

[0051] Count the number of low-level video frames. If the number of low-level video frames is less than one-third of the total number of video frames, mark the intermediate and high-level video frames as keyframes for gesture recognition. If the number of low-level video frames reaches or exceeds one-third of the total number of video frames, delete the extracted video frames and re-capture the video.

[0052] The gesture key points in each key frame are obtained in the preset XY plane rectangular coordinate system, and the coordinates of the gesture key points are obtained. The finger spacing is calculated based on the coordinates of the gesture key points, and the finger direction information is extracted. The finger spacing and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the hand position comprehensive coefficient. The hand position comprehensive coefficient is input into the pre-established gesture recognition model, and the gesture recognition result is output.

[0053] Beneficial effects of the present invention:

[0054] The present invention collects gesture video data through a video acquisition module, then uses a video processing and analysis module to calculate a video coding determination coefficient, and analyzes whether the video meets the coding requirements by setting a threshold comparison method, and then performs encoding and decoding processes on the video that meets the requirements through the encoding module and the decoding module to obtain the original video data, and then the video extraction and analysis module extracts and calculates the original video data to obtain the internal features of the video, and performs quality calculation to obtain the video frame quality coefficient, and rates the video frame quality by setting a threshold and performing a difference comparison method to screen out key frames, and then the gesture recognition module processes the key frames and comprehensively calculates the hand position to obtain the hand position comprehensive coefficient, and then recognizes the gesture through the gesture recognition model to obtain the gesture recognition result, thereby realizing the ability to perform gesture recognition after encoding and decoding various videos, and can improve the gesture recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0056] Figure 1 Schematic diagram of the system structure of the present invention;

[0057] Figure 2It is a schematic diagram of the video encoding and decoding process of the present invention;

[0058] Figure 3 It is a schematic flow chart of the method of the present invention;

[0059] Figure 4 This is a schematic diagram of transmission and playback in this embodiment. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] Example 1:

[0062] The following is an introduction to the relevant terms involved in the embodiments of this application:

[0063] WebRTC (Web Real-Time Communications) is a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between browsers without the need for an intermediary, enabling the transmission of video and / or audio streams, or any other arbitrary data. The standards included in WebRTC enable peer-to-peer data sharing and conference calls without the need to install any plugins or third-party software.

[0064] Video encoding: Video encoding is the process of converting an original video file into another format through compression technology. The most important codec standards for video streaming include ITU's H.261, H.263, and H.264, the Moving Picture Experts Group's M-JPEG, and the ISO Moving Picture Experts Group's MPEG series of standards.

[0065] Gesture recognition is a topic in computer science and language technology that aims to identify human gestures using mathematical algorithms. Gestures can originate from any body movement or state, but are typically derived from the face or hands. Current focus in this field includes emotion recognition from facial and gesture recognition. Users can use simple gestures to control or interact with devices without touching them. Recognition of posture, gait, and human behavior is also a topic of gesture recognition technology. Gesture recognition can be seen as a way for computers to understand human body language, thus building a richer bridge between machines and humans than primitive text user interfaces or even GUIs (graphical user interfaces).

[0066] Convolutional Neural Networks: Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are one of the leading algorithms for deep learning. Convolutional neural networks possess representation learning capabilities and can perform shift-invariant classification of input information based on their hierarchical structure. Hence, they are also known as shift-invariant artificial neural networks (SIANNs).

[0067] Research on convolutional neural networks began in the 1980s and 1990s, with time delay networks and LeNet-5 being the earliest convolutional neural networks. After the 21st century, with the introduction of deep learning theory and the improvement of numerical computing equipment, convolutional neural networks have developed rapidly and have been applied to fields such as computer vision and natural language processing.

[0068] Convolutional neural networks are modeled after biological visual perception mechanisms and can perform both supervised and unsupervised learning. The shared convolution kernel parameters within the hidden layer and the sparsity of inter-layer connections enable convolutional neural networks to learn grid-like topology features, such as pixels and audio, with minimal computational effort, with stable results and without requiring additional feature engineering for the data.

[0069] like Figure 1 As shown, a gesture recognition system based on video coding includes:

[0070] Video acquisition module, video processing and analysis module, encoding module, decoding module, video extraction and analysis module, gesture recognition module and server;

[0071] The video acquisition module is used to collect gesture video data and send the collected gesture video data to the video processing and analysis module for processing;

[0072] The video processing and analysis module is used to process and analyze the received gesture video data. Specifically, the processing and analysis process of the video processing module includes the following steps:

[0073] Extracting video features from the gesture video data to obtain video feature-related data, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data;

[0074] Mark the video feature related data, where the resolution feature data is marked as Bi, the frame rate feature data is marked as Zi, and the segmentation feature data is marked as Gi;

[0075] The coding decision calculation is performed using the marked video feature related data to obtain the video coding decision coefficient, where the coding decision calculation formula is as follows:

[0076]

[0077] Where Pmi is the video coding determination coefficient, B0 is the standard resolution feature coefficient, Z0 is the standard frame rate feature coefficient, G0 is the standard segmentation feature coefficient, k1 is the resolution correlation coefficient, k2 is the frame rate correlation coefficient, k3 is the segmentation correlation coefficient, and R is the preset correlation coefficient;

[0078] The calculated video coding determination coefficient is used to perform video coding analysis. Specifically, the video coding analysis process is as follows:

[0079] Set the video coding determination threshold Pm0, compare the video coding determination coefficient Pmi with the video coding determination threshold Pm0, and determine whether the video meets the coding requirements based on the comparison result:

[0080] If Pmi≥Pm0, the video processing and analysis module determines that the video meets the encoding requirements at this time, and the video processing and analysis module sends the encoding signal to the encoding module;

[0081] If Pmi<Pm0, the video processing and analysis module determines that the video does not meet the encoding requirements at this time, and deletes the video corresponding to the video encoding determination coefficient;

[0082] After receiving the coded signal sent by the video processing and analysis module, the coding module obtains the video stream corresponding to the gesture video data in the video acquisition module and then encodes it;

[0083] Specifically, the present invention will be further described below through examples:

[0084] The encoding process of the encoding module in this embodiment is as follows:

[0085] The device operates according to the received protocol, encodes the captured images into a certain format, such as H.264 or H.265, and transmits them to the streaming service via the network. The transmission process often uses a certain network protocol, the most common of which is the RTP protocol.

[0086] After connecting to the device collector through the network, the streaming media server receives data sent by the image capture device. After receiving the data from the device, it processes it using streaming media distribution technology and distributes it into streams using various transmission protocols, including common ones such as RTMP, RTSP, FLV, and WebRTC. This solution focuses on distributing streams using the WebRTC transmission protocol. Different transmission protocols store the same image data processed by the device collector.

[0087] After processing, the streaming server opens a series of network ports and waits for the required client to connect.

[0088] Chrome refers to Google Chrome, which is a general term for all common browsers. The browser (front-end) obtains the link address of WebRTC through an HTTP request.

[0089] After the front-end receives the WebRTC address returned by the HTTP request, it connects to the streaming server using the WebRTC protocol. The connection is successful.

[0090] The streaming server sends data and the browser receives data. The two have a direct channel for transmitting data to each other, DataChannel. The streaming server sends the H.265 data obtained from the frame grabber to the connected browser through the DataChannel.

[0091] Send the encoded video stream to the decoding module for decoding;

[0092] The decoding module decodes the encoded video stream after receiving it from the encoding module;

[0093] Specifically, the present invention will be further described below through examples:

[0094] The decoding process of the decoding module:

[0095] When the browser receives data from the DataChannel and detects that it is encoded in the H.265 format, it uses MSE technology to decode the data using hardware technology. This decodes the H.265 data into raw video data. The front-end player receives the raw data decoded by MSE and renders it into a video using the front-end player's rendering technology.

[0096] The decoding module sends the original video data to the video extraction and analysis module for video extraction.

[0097] The video extraction and analysis module is used to extract video from the original video data. Specifically, the extraction process of the video extraction and analysis module includes the following steps:

[0098] Performing video interception on the original video data to obtain multiple video frame data, performing feature extraction on each video frame in the multiple video frame data after the video interception to obtain video internal features, wherein the video internal features include position feature data, motion feature data and shape feature data;

[0099] In this embodiment, video capture of the original video data can be performed by using command line tools, programming language interfaces, and specialized software and tools;

[0100] Mark the internal features of the video, where the position feature data is marked as Wj, the motion feature data is marked as Yj, and the shape feature data is marked as Xj, where j is the number of internal features of the video, and j=1, 2, 3, ..., m, and m is the total number of internal features of the video;

[0101] The quality of the video frame is calculated using the marked internal features to obtain the video frame quality coefficient. Specifically, the calculation process of the video frame quality coefficient is as follows:

[0102]

[0103] Where Szj is the video frame quality coefficient, T1 is the position feature influence coefficient, T2 is the motion feature influence coefficient, T3 is the shape feature influence coefficient, X0 is the preset annotation shape feature coefficient, and α is the preset shape ratio coefficient;

[0104] Set the video frame quality coefficient threshold Sz0, use the video frame quality coefficient Szj to make a difference with the video frame quality coefficient threshold Sz0, get the difference Cj, set the difference threshold C0, perform a proportional judgment on the difference Cj and the difference threshold C0, and rate the video frame quality according to the judgment result. The specific process is as follows:

[0105] like The video frame quality is determined to be a low-level video frame;

[0106] like The video frame quality is determined to be a medium-level video frame;

[0107] like The video frame quality is determined to be a high-quality video frame;

[0108] Count the number of low-level video frames, denoted as q, if This means that the number of low-level video frames is less than one-third of the total number of video frames, indicating that the extracted video frames meet the recognition standard. The intermediate and high-level video frames are marked as key frames and sent to the gesture recognition module for gesture recognition.

[0109] like This means that the number of low-level video frames reaches or exceeds one-third of the total number of video frames, indicating that the extracted video frames do not meet the recognition standard number. The extracted video frames are deleted and a re-collection signal is sent to the video collection module. The video collection module re-collects the video until until;

[0110] The gesture recognition module is used to perform gesture recognition on key frames. Specifically, the recognition process of the gesture recognition module includes the following steps:

[0111] Establishing an XY plane rectangular coordinate system through the gesture recognition module, and using the coordinate acquisition unit of the gesture recognition module to obtain gesture key points in each key frame in the plane rectangular coordinate system, including a first key point, a second key point, a third key point, a fourth key point, and a fifth key point;

[0112] In this embodiment, each key point represents the coordinates of the fingertip position of a finger. The first key point is the coordinates of the thumb tip, the second key point is the coordinates of the index finger tip, the third key point is the coordinates of the middle finger tip, the fourth key point is the coordinates of the ring finger tip, and the fifth key point is the coordinates of the little finger tip.

[0113] Among them, the first key point is marked as A1(X 1u , Y 1u ), the second key point is marked as A2(X 2u , Y 2u ), the third key point is marked as A3(X 3u , Y 3u ), the fourth key point is marked as A4(X 4u , Y 4u ); the fifth key point is marked as A5(X 5u , Y 5u ); where u is the number of each key frame, and u = 1, 2, 3, ..., V; V is the total number of key frames;

[0114] The finger distance is calculated using the coordinates of the first key point, the second key point, the third key point, the fourth key point, and the fifth key point, where the distance between one and two fingers is marked as a 12u , the distance between two and three fingers is marked as a 23u , the distance between three and four fingers is marked as a 34u , the distance between four and five fingers is marked as a 45u ;

[0115] Extract the direction information of each finger;

[0116] The finger distance and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the comprehensive hand position coefficient. Specifically, the calculation process is as follows:

[0117]

[0118] Where sub is the comprehensive coefficient of hand position, l1, l2, l3, and l4 are all preset direction correlation coefficients, and β is the preset position proportional coefficient;

[0119] The hand position comprehensive coefficient sub is input into a pre-established gesture recognition model, and the gesture recognition result is output; thereby, the gesture recognition result is displayed so that the gesture action can be obtained;

[0120] Wherein, the pre-established gesture recognition model is based on artificial intelligence model training;

[0121] The gesture recognition model is trained based on the artificial intelligence model. The specific process is as follows:

[0122] Integrating and acquiring standard hand-related data through a server, wherein the standard hand-related data includes standard finger spacing data, standard finger position coordinate data, and standard finger direction data;

[0123] The artificial intelligence model is trained using standard hand-related data to obtain and store a gesture recognition model; wherein the artificial intelligence model includes a deep convolutional neural network model and an RBF neural network model.

[0124] In this embodiment, the preset direction correlation coefficient is obtained based on the direction information of each finger;

[0125] Specifically, the present invention will be further described below through examples:

[0126] Embodiment 2: A gesture recognition method based on video coding, the method comprising the following steps:

[0127] Acquiring gesture video data, performing video feature extraction on the gesture video data to obtain video feature-related data, and performing coding determination calculation using the video feature-related data to obtain a video coding determination coefficient, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data;

[0128] Set the video coding determination threshold, compare the video coding determination coefficient with the video coding determination threshold, and determine whether the video meets the coding requirements based on the comparison result. If not, delete the corresponding video. If it meets the requirements, perform the encoding and decoding process on the video, and obtain the original video data after decoding;

[0129] Performing video interception on original video data to obtain multiple video frame data, performing feature extraction on each video frame in the multiple video frame data after the video interception to obtain video internal features, and performing quality calculation using the video internal features to obtain a video frame quality coefficient, wherein the video internal features include position feature data, motion feature data, and shape feature data;

[0130] Setting a video frame quality coefficient threshold, performing a difference calculation between the video frame quality coefficient and the video frame quality coefficient threshold to obtain a difference, setting a difference threshold, performing a proportional determination between the difference and the difference threshold, and rating the video frame quality according to the determination result, wherein the rated video levels include: low-level video frame, medium-level video frame, and high-level video frame;

[0131] Count the number of low-level video frames. If the number of low-level video frames is less than one-third of the total number of video frames, mark the intermediate and high-level video frames as keyframes for gesture recognition. If the number of low-level video frames reaches or exceeds one-third of the total number of video frames, delete the extracted video frames and re-capture the video.

[0132] The gesture key points in each key frame are obtained in the preset XY plane rectangular coordinate system, and the coordinates of the gesture key points are obtained. The finger spacing is calculated based on the coordinates of the gesture key points, and the finger direction information is extracted. The finger spacing and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the hand position comprehensive coefficient. The hand position comprehensive coefficient sub is input into the pre-established gesture recognition model, and the gesture recognition result is output.

[0133] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0134] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which executes the above method when executed by a processor. The storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0135] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0136] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present disclosure. Various changes and improvements may be made to the present disclosure without departing from the spirit and scope of the present disclosure, and such changes and improvements shall fall within the scope of the present disclosure.

Claims

1. A gesture recognition system based on video coding, characterized in that: include: Video acquisition module: used to collect gesture video data and send the collected gesture video data to the video processing and analysis module for processing; Video processing and analysis module: used to extract video features from gesture video data to obtain video feature-related data, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data; use the video feature-related data to perform coding determination calculation to obtain a video coding determination coefficient; set a video coding determination threshold; compare the video coding determination coefficient with the video coding determination threshold; and determine whether the video meets the coding requirements based on the comparison result. If not, the corresponding video is deleted; if it meets, a coding signal is sent to the coding module; Encoding module: used to obtain the video stream corresponding to the gesture video data in the video acquisition module, then encode it to obtain the encoded video stream, and send the encoded video stream to the decoding module; Decoding module: used to decode the encoded video stream to obtain the original video data; Video extraction and analysis module: used to perform video interception on the original video data to obtain multiple video frame data, perform feature extraction on each video frame in the multiple video frame data after the video interception, obtain video internal features, wherein the video internal features include position feature data, motion feature data and shape feature data, perform quality calculation using the marked video internal features to obtain a video frame quality coefficient, set a video frame quality coefficient threshold, use the video frame quality coefficient and the video frame quality coefficient threshold to make a difference to obtain a difference, set a difference threshold, perform a ratio determination between the difference and the difference threshold, and rate the video frame quality according to the determination result, wherein the rated video levels include: low-level video frames, medium-level video frames and high-level video frames; Count the number of low-level video frames. If the number of low-level video frames is less than one-third of the total number of video frames, mark the intermediate and high-level video frames as key frames and send the key frames to the gesture recognition module. If the number of low-level video frames reaches or exceeds one-third of the total number of video frames, delete the extracted video frames and recapture the video. Gesture recognition module: used to establish an XY plane rectangular coordinate system, use the coordinate acquisition unit of the gesture recognition module to obtain the gesture key points in each key frame in the plane rectangular coordinate system, obtain the coordinates of the gesture key points, calculate the finger spacing based on the gesture key point coordinates, and extract the finger direction information. Use the finger spacing and finger direction information to perform a comprehensive calculation of the hand position to obtain the hand position comprehensive coefficient. The hand position comprehensive coefficient is input into a pre-established gesture recognition model to output the gesture recognition result; Server: Used to integrate and obtain standard hand-related data required for training pre-established gesture recognition models.

2. The gesture recognition system based on video coding according to claim 1, characterized in that: The calculation process of the video processing and analysis module: Mark the video feature related data, where the resolution feature data is marked as Bi, the frame rate feature data is marked as Zi, and the segmentation feature data is marked as Gi; The coding decision calculation is performed using the marked video feature related data. The coding decision calculation formula is as follows: Where Pmi is the video coding determination coefficient, B0 is the standard resolution feature coefficient, Z0 is the standard frame rate feature coefficient, G0 is the standard segmentation feature coefficient, k1 is the resolution correlation coefficient, k2 is the frame rate correlation coefficient, k3 is the segmentation correlation coefficient, and R is the preset correlation coefficient.

3. The gesture recognition system based on video coding according to claim 2, characterized in that: The analysis process of the video processing and analysis module: Set the video coding determination threshold Pm0, and compare the video coding determination coefficient Pmi with the video coding determination threshold Pm0 to determine: If Pmi≥Pm0, it is determined that the video meets the encoding requirements and the encoding signal is sent to the encoding module; If Pmi<Pm0, it is determined that the video does not meet the encoding requirements at this time, and the video corresponding to the video encoding determination coefficient is deleted.

4. The gesture recognition system based on video coding according to claim 1, characterized in that: The calculation process of the video extraction and analysis module: Mark the internal features of the video, where the position feature data is marked as Wj, the motion feature data is marked as Yj, and the shape feature data is marked as Xj, where j is the number of internal features of the video, and j=1, 2, 3, ..., m, and m is the total number of internal features of the video; The quality of the video frame is calculated using the marked internal features to obtain the video frame quality coefficient. Specifically, the calculation process of the video frame quality coefficient is as follows: Where Szj is the video frame quality coefficient, T1 is the position feature influence coefficient, T2 is the motion feature influence coefficient, T3 is the shape feature influence coefficient, X0 is the preset annotation shape feature coefficient, and α is the preset shape ratio coefficient.

5. The gesture recognition system based on video coding according to claim 4, characterized in that: The analysis process of the video extraction and analysis module: Set the video frame quality coefficient threshold Sz0, use the video frame quality coefficient Szj to make a difference with the video frame quality coefficient threshold Sz0, obtain the difference Cj, set the difference threshold C0, and perform a proportional determination between the difference Cj and the difference threshold C0: like Determining the video frame quality as low-level video frame; like The video frame quality is determined to be medium-level; like Determine the video frame quality as high-quality video frame; Count the number of low-level video frames, denoted as q, if At this point, the extracted video frames meet the recognition standard number, the intermediate video frames and the advanced video frames are marked as key frames, and the key frames are sent to the gesture recognition module for gesture recognition; like If the extracted video frames do not meet the recognition standard number, the extracted video frames will be deleted and a re-collection signal will be sent to the video collection module. until.

6. The gesture recognition system based on video coding according to claim 1, characterized in that: The gesture key points in the gesture recognition module include a first key point, a second key point, a third key point, a fourth key point and a fifth key point; Among them, the first key point is marked as A1(X 1u , Y 1u ), the second key point is marked as A2(X 2u , Y 2u ), the third key point is marked as A3(X 3u , Y 3u ), the fourth key point is marked as A4(X 4u , Y 4u ); the fifth key point is marked as A5(X 5u , Y 5u ), where u is the number of each key frame, and u = 1, 2, 3, ..., V; V is the total number of key frames.

7. The gesture recognition system based on video coding according to claim 6, characterized in that: The gesture recognition module calculates the finger distance using the coordinates of the first key point, the second key point, the third key point, the fourth key point, and the fifth key point, where the distance between one and two fingers is marked as a. 12u , the distance between two and three fingers is marked as a 23u , the distance between three and four fingers is marked as a 34u , the distance between four and five fingers is marked as a 45u ; Extract the direction information of each finger; The finger distance and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the comprehensive hand position coefficient: Where sub is the comprehensive coefficient of hand position, l1, l2, l3, and l4 are all preset direction correlation coefficients, and β is the preset position proportional coefficient.

8. The gesture recognition system based on video coding according to claim 7, characterized in that: The gesture recognition model pre-established by the gesture recognition module is trained based on an artificial intelligence model; Integrating and acquiring standard hand-related data through a server, wherein the standard hand-related data includes standard finger spacing data, standard finger position coordinate data, and standard finger direction data; The artificial intelligence model is trained using standard hand-related data to obtain and store the gesture recognition model.

9. The gesture recognition system based on video coding according to claim 1, characterized in that: The decoding module can directly decode the video stream in H.265 format encoded by the encoding module.

10. A gesture recognition method based on video coding, characterized in that: The method comprises the following steps: Acquiring gesture video data, performing video feature extraction on the gesture video data to obtain video feature-related data, and performing coding determination calculation using the video feature-related data to obtain a video coding determination coefficient, wherein the video feature-related data includes: resolution feature data, frame rate feature data, and segmentation feature data; Set the video coding determination threshold, compare the video coding determination coefficient with the video coding determination threshold, and determine whether the video meets the coding requirements based on the comparison result. If not, delete the corresponding video. If it meets the requirements, perform the encoding and decoding process on the video, and obtain the original video data after decoding; Performing video interception on original video data to obtain multiple video frame data, performing feature extraction on each video frame in the multiple video frame data after the video interception to obtain video internal features, and performing quality calculation using the video internal features to obtain a video frame quality coefficient, wherein the video internal features include position feature data, motion feature data, and shape feature data; Setting a video frame quality coefficient threshold, performing a difference calculation between the video frame quality coefficient and the video frame quality coefficient threshold to obtain a difference, setting a difference threshold, performing a proportional determination between the difference and the difference threshold, and rating the video frame quality according to the determination result, wherein the rated video levels include: low-level video frame, medium-level video frame, and high-level video frame; Count the number of low-level video frames. If the number of low-level video frames is less than one-third of the total number of video frames, mark the intermediate and high-level video frames as keyframes for gesture recognition. If the number of low-level video frames reaches or exceeds one-third of the total number of video frames, delete the extracted video frames and re-capture the video. The gesture key points in each key frame are obtained in the preset XY plane rectangular coordinate system, and the coordinates of the gesture key points are obtained. The finger spacing is calculated based on the coordinates of the gesture key points, and the finger direction information is extracted. The finger spacing and finger direction information are used to perform a comprehensive calculation of the hand position to obtain the hand position comprehensive coefficient. The hand position comprehensive coefficient is input into the pre-established gesture recognition model, and the gesture recognition result is output.

Images