A lightweight sign language recognition method, system, device and medium
By connecting the front-end and back-end networks and deploying them on portable devices and cloud servers, the existing sign language recognition methods are solved, and real-time sign language recognition on portable devices is realized and recognition accuracy is improved.
Patent Information
- Application Number
- CN202410162427.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-02-05
AI Technical Summary
The feature extraction neural network used by existing sign language visual recognition methods when training models is complex in structure, large calculations, and long inference time, making it difficult to realize real-time sign language recognition on portable devices.
A lightweight sign language recognition method is proposed, which forms an overall recognition network through the connection processing of the front-end network and the back-end network, and is deployed on portable devices and cloud servers. The cloud server is used to accelerate inference recognition and improve recognition speed and accuracy.
It realizes the rapidity and real-time nature of real-time sign language translation on portable devices, improves the accuracy and speed of sign language recognition, and meets the real-time sign language translation needs on portable devices.
Smart Images

Figure CN117994846B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a lightweight sign language recognition method, system, device and medium. Background Art
[0002] In the related art, sign language recognition is divided into data sensing gloves and sign language visual recognition. Among them, sign language visual recognition uses computer vision technology to capture the user's hand features to achieve sign language recognition and translation. However, the feature extraction neural network structure used in the sign language visual recognition method when training the model is relatively complex, the amount of calculation is large, and the reasoning time is long, which has great limitations when used on portable devices. In summary, the technical problems existing in the related art need to be improved. Summary of the invention
[0003] The main purpose of the embodiments of the present application is to propose a lightweight sign language recognition method, system, device and medium, which can improve the convenience and real-time performance of sign language recognition.
[0004] To achieve the above object, an embodiment of the present application provides a lightweight sign language recognition method, the method comprising:
[0005] Get a training image set;
[0006] The training image set is screened and cropped by a front-end network to obtain a conversion image set;
[0007] Inputting the converted image set into an image classification network for training processing to obtain a backend network;
[0008] Connecting the front-end network and the back-end network to obtain an overall identification network;
[0009] The overall recognition network is converted into an accelerated inference framework and is respectively deployed on a portable device and a cloud server to perform sign language recognition processing on the sign language action to be recognized, thereby obtaining a sign language recognition result.
[0010] In some embodiments, the front-end network includes a posture estimation network and a hand node recognition network, and the front-end network is used to filter and crop the training image set to obtain a conversion image set, including:
[0011] Performing key point detection processing on each training image in the training image set through the posture estimation network to obtain upper body posture nodes and node scores;
[0012] The training image is screened according to the node score, and the screened image is intercepted to obtain a hand rectangular frame;
[0013] Performing node recognition processing on the hand rectangular frame through the hand node recognition network to obtain the hand joint positions;
[0014] The training images are transformed according to the upper body posture nodes and the hand joint positions to obtain a transformed image set.
[0015] In some embodiments, the converting process of the training image according to the upper body posture nodes and the hand joint positions to obtain a converted image set includes:
[0016] Get the black background image;
[0017] According to the upper body posture nodes and the hand joint positions, corresponding positions in the black background image are illuminated and connected to obtain a skeleton image;
[0018] The boundary position of the skeleton image is intercepted to obtain a conversion image set.
[0019] In some embodiments, the step of inputting the converted image set into an image classification network for training to obtain a backend network includes:
[0020] Initializing the weight parameters of the image classification network to obtain an initial network;
[0021] The converted image set is normalized and horizontally randomly flipped to obtain a training set and a validation set;
[0022] The model parameters of the initial network are updated according to the training set and the validation set to obtain a backend network.
[0023] In some embodiments, the steps of performing sign language recognition processing on the sign language action to be recognized, respectively deployed on the portable device and the cloud server, to obtain a sign language recognition result, include:
[0024] Performing image acquisition processing on the sign language action to be recognized by the portable device to obtain a sign language image to be recognized;
[0025] Establishing a connection channel between the portable device and the cloud server;
[0026] Selecting an inference mode according to the delay of the connection channel to perform inference recognition processing on the sign language image to be recognized, and obtaining an inference recognition result;
[0027] The inference recognition result is evaluated and processed in combination with the overall recognition network to obtain a sign language recognition result.
[0028] In some embodiments, the step of evaluating the inference recognition result in combination with the overall recognition network to obtain the sign language recognition result includes:
[0029] Selecting and recording the training image set to obtain a sign language action video;
[0030] Inputting the sign language action video frame by frame into the overall recognition network for recognition processing, and performing accuracy calculation processing on the recognition results to obtain an accuracy standard;
[0031] The inference recognition result is screened according to the accuracy standard, and judged in combination with the inference speed and frame rate display of the inference recognition result to obtain the sign language recognition result.
[0032] In some embodiments, obtaining a training image set includes:
[0033] Acquire sign language vocabulary, and record a sign language action video according to the sign language vocabulary;
[0034] The sign language action video is subjected to key frame interception processing to obtain a training image set.
[0035] To achieve the above object, another aspect of the embodiment of the present application provides a lightweight sign language recognition system, the system comprising:
[0036] The first module is used to obtain a training image set;
[0037] The second module is used to screen and crop the training image set through a front-end network to obtain a conversion image set;
[0038] The third module is used to input the converted image set into the image classification network for training processing to obtain a backend network;
[0039] The fourth module is used to connect the front-end network and the back-end network to obtain an overall identification network;
[0040] The fifth module is used to accelerate the inference framework conversion processing of the overall recognition network, and is deployed on the portable device and the cloud server respectively to perform sign language recognition processing on the sign language action to be recognized to obtain the sign language recognition result.
[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.
[0042] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0043] The embodiments of the present application include at least the following beneficial effects: the present application provides a lightweight sign language recognition method, system, device and medium, which obtains an overall recognition network by connecting the front-end network and the back-end network; then the overall recognition network is converted into an accelerated reasoning framework, and is respectively deployed on a portable device and a cloud server to perform sign language recognition processing on the sign language action to be recognized, thereby obtaining a sign language recognition result; it can meet the requirements of rapidity and real-time performance of real-time sign language translation on a portable device, and at the same time, utilize a cloud server for accelerated reasoning recognition to improve the recognition rate and speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a flow chart of a lightweight sign language recognition method provided by an embodiment of the present application;
[0045] Figure 2 is a flow chart of a sign language image conversion provided by an embodiment of the present application;
[0046] Figure 3 This is a flow chart of an overall recognition network for image recognition provided by an embodiment of the present application;
[0047] Figure 4 is a flow chart for identifying a reasoning mode provided in an embodiment of the present application;
[0048] Figure 5 It is a specific implementation flow chart provided by the embodiment of the present application;
[0049] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.
[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".
[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.
[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0054] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.
[0055] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0057] Among related technologies, sign language research involves multiple fields and interdisciplinary subjects. At present, the field of sign language recognition is booming with two mainstream research directions: data sensing gloves and sign language visual recognition. The former uses sensors to collect motion data for sign language recognition and translation, while the latter uses computer vision technology to capture user hand features to achieve sign language recognition and translation.
[0058] For example, data gloves are used to collect the user's hand motion change data, extract its motion features and perform gesture recognition. Gesture recognition is performed by fusing texture features, geometric features and convolutional neural networks, thereby improving the recognition rate of sign language. A method combining convolutional neural networks and long short-term memory networks is used to recognize dynamic gesture sequences, with an average recognition rate of 92.5%. A volumetric neural network is used to train and predict ASL data sets with an accuracy of 94.7%. A static gesture recognition method based on convolutional neural networks is used. After image acquisition, image recognition, data processing, CNN modeling, parameter training and other steps, the accuracy of the model is verified by the test set to reach 92.19%. In view of the fact that complex backgrounds will have a certain degree of influence on gesture recognition, a gesture recognition scheme based on the AlexNet optimization model is proposed, and the recognition accuracy of the scheme can reach 93%. The hand shape features of sign language are extracted using convolutional neural networks, and continuous dynamic sign language is recognized using trajectory normalization algorithms and long short-term memory networks. The algorithm can recognize sentences composed of 47 commonly used sign language words. However, the feature extraction neural network structure used in the above research content when training the model is relatively complex, the amount of calculation is large, the reasoning time is long, and it has great limitations when used on portable devices.
[0059] In view of this, a lightweight sign language recognition method, system, device and medium are provided in the embodiments of the present application. The scheme obtains an overall recognition network by connecting the front-end network and the back-end network; then the overall recognition network is converted into an accelerated reasoning framework, and is respectively deployed on a portable device and a cloud server to perform sign language recognition processing on the sign language actions to be recognized, so as to obtain a sign language recognition result; it can meet the requirements of rapidity and real-time of real-time sign language translation on a portable device, and at the same time can use the cloud server for accelerated reasoning recognition to improve the recognition rate and speed.
[0060] The lightweight sign language recognition method provided in the embodiment of the present application relates to the field of computer technology. The lightweight sign language recognition method provided in the embodiment of the present application can be applied to portable devices, can also be applied to servers, and can also be software running in portable devices or servers. In some embodiments, the portable device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in the blockchain network; the software can be an application that implements the lightweight sign language recognition method, etc., but is not limited to the above forms.
[0061] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0062] Figure 1 is an optional flowchart of a lightweight sign language recognition method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.
[0063] Step S101, obtaining a training image set;
[0064] Step S102, screening and cropping the training image set through a front-end network to obtain a conversion image set;
[0065] Step S103, inputting the converted image set into an image classification network for training processing to obtain a backend network;
[0066] Step S104, connecting the front-end network and the back-end network to obtain an overall identification network;
[0067] Step S105, performing accelerated inference framework conversion processing on the overall recognition network, and deploying it on the portable device and the cloud server respectively to perform sign language recognition processing on the sign language action to be recognized, to obtain a sign language recognition result.
[0068] In the steps S101 to S105 shown in the embodiment of the present application, the training image set is screened and cropped by the front-end network, so that the sign language images in the training image set are converted into the format processed by the image classification network to obtain the converted image set. Then the converted image set is input into the image classification network for training, the trained image classification network is used as the back-end network, and the front-end network and the back-end network are connected to obtain the overall recognition network. Finally, the overall recognition network is converted into a framework and form that can accelerate reasoning, and is deployed on the portable device and the cloud server respectively to perform sign language recognition processing on the sign language action to be recognized, and obtain the sign language recognition result. In view of the large amount of calculation of the existing sign language technology, it is difficult to take into account the requirements of portability and real-time performance. The embodiment of the present application takes the design and implementation of a lightweight sign language recognition system as the main goal, compares and studies a variety of machine vision models, studies their precision and accuracy, and comprehensively considers the amount of calculation and the speed of calculation to meet the requirements of rapidity and real-time performance of sign language translation in real time on portable devices. At the same time, the cloud server is used to accelerate reasoning recognition, which improves the accuracy and speed of sign language recognition.
[0069] In step S101 of some embodiments, obtaining a training image set includes:
[0070] Acquire sign language vocabulary, and record a sign language action video according to the sign language vocabulary;
[0071] The sign language action video is subjected to key frame interception processing to obtain a training image set.
[0072] In an embodiment of the present application, a corresponding sign language video is obtained by recording a number of sign language words, and then the video is intercepted and processed to form a training image set. In a feasible embodiment, by obtaining 20 commonly used sign language words, different personnel record sign language action videos consisting of 3-5 sign language words at different times, and for each video, key frame images that are significantly different from other sign language actions are intercepted and retained, and the key frames in each type of sign language action video that are significantly different from other actions are intercepted and retained until more than 200 images of each type of sign language action are obtained after retention to obtain a training image set. The embodiment of the present application provides training data for subsequent sign language recognition processing by recording sign language words and intercepting key frames.
[0073] In step S102 of some embodiments, the front-end network includes a posture estimation network and a hand node recognition network, and the front-end network is used to filter and crop the training image set to obtain a conversion image set, including:
[0074] Performing key point detection processing on each training image in the training image set through the posture estimation network to obtain upper body posture nodes and node scores;
[0075] The training image is screened according to the node score, and the screened image is intercepted to obtain a hand rectangular frame;
[0076] Performing node recognition processing on the hand rectangular frame through the hand node recognition network to obtain the hand joint positions;
[0077] The training images are transformed according to the upper body posture nodes and the hand joint positions to obtain a transformed image set.
[0078] In an embodiment of the present application, the front-end network includes a posture estimation network and a hand node recognition network, wherein the embodiment of the present application selects a pre-trained posture estimation network and a hand node recognition network with a small amount of calculation, a fast inference speed and a high accuracy. The posture estimation network can use a fast and accurate MoveNet model, which can detect 17 key points of the body. The Lightning model, one of the variants of this model, is used for delay-critical applications and can run on most desktops, laptops and mobile phones. The hand node recognition model can use an image convolution network based on SqueezeNet, which can detect 21 nodes of the hand. SqueezeNet is a network that maximizes the computing speed while ensuring that the model accuracy is not reduced. It can achieve an effect similar to that of the AlexNet model on the ImageNet data set, but the parameters are 50 times less than the AlexNet model. The embodiment of the present application can also be combined with the model compression technology Deep Compression to make the model file of the hand recognition network 510 times smaller than the AlexNet model. The embodiment of the present application performs key point detection processing on each training image in the training image set through the posture estimation network, and detects and obtains 17 key point positions and their scores. First, the 11 nodes of the upper body and their scores are retained to obtain the upper body posture nodes and node scores. The training images are screened and processed according to the node scores, and the screened images are intercepted and processed to obtain the hand rectangular frame. Specifically, the detection scores of the three nodes (shoulder, elbow, and wrist) of the hands are judged to be higher than the threshold value of 0.3. After the judgment is passed, the shoulder, elbow, and wrist coordinates corresponding to the hand that meet the judgment requirements are set to the wrist elbow ratio of 0.3 to determine the hand center, that is, 3 / 10 of the line from the wrist to the elbow is designated as the hand center, and then the wrist elbow distance and 0.9 times the shoulder elbow distance are compared. The maximum value of the two is taken as the half-side length of the square frame, and then the four-point coordinates of the rectangular frame are calculated to obtain the hand rectangular frame. The obtained hand rectangular frame is then subjected to node recognition processing by the hand node recognition network to obtain the hand joint position. Specifically, the hand rectangular frame is reshaped into a size of 256x256, the pixel matrix of the hand rectangular frame is normalized to the range of -1 to 1, and the pre-trained hand node recognition model is put into it to identify the 21 node positions of the hand, and the hand joint positions are obtained, so that the 21 node positions of the hand can be inverted according to the size and position of the rectangular frame to their corresponding positions in the original sign language image containing the human body. Finally, the training image is converted according to the upper body posture nodes and the hand joint positions to obtain a converted image set. The embodiment of the present application screens and crops the training image set through the front-end network to obtain a converted image set, which provides a data basis for the subsequent image classification network to perform recognition.
[0079] In some embodiments, the converting process of the training image according to the upper body posture nodes and the hand joint positions to obtain a converted image set includes:
[0080] Get the black background image;
[0081] According to the upper body posture nodes and the hand joint positions, corresponding positions in the black background image are illuminated and connected to obtain a skeleton image;
[0082] The boundary position of the skeleton image is intercepted to obtain a conversion image set.
[0083] In an embodiment of the present application, a black background image is obtained, the size of which is the same as that of the training image, and then the upper body posture nodes and hand node positions extracted in the above steps are used to find the corresponding positions in the black background image, light them up, and connect them in sequence to obtain a skeleton image. At this time, it is necessary to cut out the core part with bright spots, perform bright spot judgment on the pixel matrix of the skeleton image, obtain the maximum and minimum horizontal and vertical coordinate values of the non-black part, add the boundary margin to obtain the boundary position of the interception box, and then intercept it to obtain the final converted image to obtain the converted image set. The boundary margin can be 15px. Figure 2 In the embodiment of the present application, the sign language image is input into the posture estimation network, the upper body nodes are intercepted according to the result of the posture estimation network, and then the valid image is retained according to the requirement that the wrist is higher than the elbow, and the scores of the three nodes of the shoulder, elbow, and wrist of one of the left and right hands are all higher than the threshold. The retained image is input into the hand node recognition network for hand node recognition, and only the upper body part is retained according to the posture estimation result. In combination with the hand node recognition result, the corresponding positions of the black background image of the same size as the original image are lit and connected to each other to form a skeleton diagram that only retains the upper body. The above skeleton diagram is cropped, and the upper, lower, left, and right bright spot boundaries are expanded according to a certain boundary margin. The image is intercepted according to the expanded range to obtain the converted training set. In the embodiment of the present application, the converted image set is obtained by converting the training image, which provides a data basis for the subsequent training of the image classification network.
[0084] In step S103 of some embodiments, the inputting the converted image set into an image classification network for training processing to obtain a backend network includes:
[0085] Initializing the weight parameters of the image classification network to obtain an initial network;
[0086] The converted image set is normalized and horizontally randomly flipped to obtain a training set and a validation set;
[0087] The model parameters of the initial network are updated according to the training set and the validation set to obtain a backend network.
[0088] In the embodiment of the present application, the converted image set is input into the image classification network for training. First, the weight parameters of the image classification network are initialized. The setting parameters of the Micronet-m3 model can be used to initialize the weight parameters in the model. Then, the images in the converted image set are cropped, standardized, and randomly flipped horizontally, and then split into a training set and a validation set. Then, the relevant hyperparameters are set: the cross entropy function (CrossEntropyLoss) is selected to calculate the loss, and the Adam method is used to update the model parameters to obtain the backend network. The update strategy is shown in the following formula:
[0089] L n =Lr0*[n / step] gamma ;
[0090] In the formula, Lr0 is the initial learning rate, is the learning rate of the mth training cycle, step is the attenuation step size, gamma is the attenuation coefficient, and [x] is the rounding function.
[0091] In the embodiment of the present application, the initial learning rate is set to Lr0=0.001, the number of images in each batch training is batch_size=16, the maximum training cycle is max_epoch=100, and the learning rate update strategy is exponentially decreasing. Figure 3 After the training of the image classification network is completed, the embodiment of the present application obtains the back-end network, and then connects the front-end network with the back-end network to obtain the overall recognition network. By inputting the image into the overall recognition network, the converted image is obtained after undergoing the image conversion process, and then inputted into the image classification network, and the semantic recognition result is obtained by output. The embodiment of the present application obtains the back-end network by training the image classification network, and then performs sign language recognition according to the back-end network, thereby improving the recognition accuracy and speed.
[0092] In step S105 of some embodiments, the processes respectively deployed on the portable device and the cloud server perform sign language recognition processing on the sign language action to be recognized to obtain a sign language recognition result include:
[0093] Performing image acquisition processing on the sign language action to be recognized by the portable device to obtain a sign language image to be recognized;
[0094] Establishing a connection channel between the portable device and the cloud server;
[0095] Selecting an inference mode according to the delay of the connection channel to perform inference recognition processing on the sign language image to be recognized, and obtaining an inference recognition result;
[0096] The inference recognition result is evaluated and processed in combination with the overall recognition network to obtain a sign language recognition result.
[0097] In the embodiment of the present application, each model in the overall recognition network is converted into a framework and form that can accelerate reasoning, and is deployed on a portable device and a cloud server respectively. Among them, the portable device can use a Raspberry Pi, the model is Raspberry Pi 4B, the camera selected is Raspberry Pi Camera Module 3, and the reasoning framework is the graph structure model (Intermediate Representation, Ir model) of Openvino. The cloud server can use the Alibaba Cloud server accelerated by the unified computing device architecture (Compute Unified Device Architecture, CUDA) with Nvidia GPU, and the reasoning framework is the engine model of TensorRT. Then, the portable device is used to obtain the image of the sign language action to be recognized, and the sign language image to be recognized is obtained by recording, shooting and other methods. A connection channel is established between the portable device and the cloud server, and the portable device establishes two socket connections (port multiplexing) with a port corresponding to the public network IP of the cloud server through the TCP protocol: one is the picture channel, and the compressed and encoded image is transmitted to the cloud server in segments, and the cloud server decodes it into a picture after receiving it. The other is the result channel, which transmits the recognition result of the cloud server back to the portable device. In the embodiment of the present application, the main process can also be set to read the camera image and put it into the queue, and the sub-process obtains the image from the queue for recognition to avoid screen freeze. Then, according to the delay of the connection channel, the inference mode is selected to perform inference recognition processing on the sign language image to be recognized to obtain the inference recognition result. Figure 4 The reasoning modes include online mode, offline mode and comprehensive mode. The online mode only performs recognition reasoning in the cloud, which is suitable for situations with low latency; the offline mode only performs recognition reasoning on the local side, which is suitable for situations with high latency. The comprehensive mode performs reasoning and recognition on the acquired images crosswise on the cloud and the local side at the same time, and integrates the results into a sequence according to the order of recognition. Finally, the reasoning and recognition results are evaluated and processed in combination with the overall recognition network to obtain the sign language recognition results. The embodiment of the present application deploys the overall recognition network on a portable device and a cloud server respectively to meet the requirements of speed and real-time sign language translation in real time on a portable device. At the same time, the cloud server is used to accelerate reasoning and recognition, and recognition is performed alternately on the online recognition terminal and the local recognition terminal to improve the recognition rate and speed.
[0098] In some embodiments, the step of evaluating the inference recognition result in combination with the overall recognition network to obtain the sign language recognition result includes:
[0099] Selecting and recording the training image set to obtain a sign language action video;
[0100] Inputting the sign language action video frame by frame into the overall recognition network for recognition processing, and performing accuracy calculation processing on the recognition results to obtain an accuracy standard;
[0101] The inference recognition result is screened according to the accuracy standard, and judged in combination with the inference speed and frame rate display of the inference recognition result to obtain the sign language recognition result.
[0102] In an embodiment of the present application, 10 of the 20 commonly used sign language words included in the training image set are selected to record new sign language action videos. The new sign language action video is placed frame by frame in the overall recognition network for recognition, and the recognition accuracy is calculated, that is, the number of correctly recognized frames divided by the total number of frames. Then the accuracy of each type of test action is averaged to obtain the average accuracy, and the average accuracy is determined as the accuracy standard. Then, according to the calculated accuracy standard and combined with the reasoning speed, the requirement of the same recognition results for 4-6 consecutive frames is set, and the results that meet the requirements in the result sequence are output, and the time used for each recognition and the corresponding frame rate are calculated at the same time. The comprehensive result output and frame rate display are used as the final evaluation result to judge the sign language recognition result and obtain the sign language recognition result.
[0103] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:
[0104] The embodiments of the present application involve computer vision recognition methods and convolutional neural network algorithms, which are applicable to daily communication scenarios between deaf-mute people and non-deaf-mute people. Compared with traditional sign language software that can only be used for daily communication between hearing-impaired people and ordinary people and in the field of public services, the embodiments of the present application can also be applied to the field of sign language education, and in the future it can be applied to human-computer interaction, virtual reality, game control and other fields. Figure 5 In the data preparation stage of the embodiment of the present application, the sign language videos corresponding to several words recorded by the team members themselves are used to form a data set, and a preliminary screening is performed. Then, in the image conversion stage, for each type of sign language image, the posture estimation network and the hand node recognition network are combined for screening and cropping. The posture estimation network is connected in series with the hand node recognition network as the front end, and the back end is connected to the trained image classification network to connect the front and back end networks. In the network prediction stage, the test set is input into the network connected to the front and back ends for prediction, and the sign language semantic recognition result is obtained. Finally, by designing the recognition mode, the network connected to the front and back ends is deployed to the portable device and the cloud server at the same time, and a corresponding reasoning recognition mode is designed to recognize the sign language movements to obtain the sign language recognition result.
[0105] The embodiment of the present application also provides a lightweight sign language recognition system, which can implement the above lightweight sign language recognition method, and the system includes:
[0106] The first module is used to obtain a training image set;
[0107] The second module is used to screen and crop the training image set through a front-end network to obtain a conversion image set;
[0108] The third module is used to input the converted image set into the image classification network for training processing to obtain a backend network;
[0109] The fourth module is used to connect the front-end network and the back-end network to obtain an overall identification network;
[0110] The fifth module is used to accelerate the inference framework conversion processing of the overall recognition network, and is deployed on the portable device and the cloud server respectively to perform sign language recognition processing on the sign language action to be recognized to obtain the sign language recognition result.
[0111] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0112] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned lightweight sign language recognition method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0113] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0114] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0115] The processor 601 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0116] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 calls and executes the lightweight sign language recognition method of the embodiment of this application;
[0117] Input / output interface 603, used to implement information input and output;
[0118] Communication interface 604, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0119] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );
[0120] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .
[0121] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned lightweight sign language recognition method is implemented.
[0122] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0123] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0124] The embodiments of the present application provide a lightweight sign language recognition method, system, device and medium, which obtain an overall recognition network by connecting a front-end network and a back-end network; then the overall recognition network is converted into an accelerated reasoning framework, and is respectively deployed on a portable device and a cloud server to perform sign language recognition processing on the sign language action to be recognized, thereby obtaining a sign language recognition result; the requirements of rapidity and real-time nature of real-time sign language translation on a portable device can be met, and at the same time, the cloud server is used to perform accelerated reasoning recognition to improve the recognition rate and speed.
[0125] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0126] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0127] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0128] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0129] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0130] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0131] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0132] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0135] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A lightweight sign language recognition method, It is characterized in that The method comprises: Get a training image set; The training image set is screened and cropped by a front-end network to obtain a conversion image set; Inputting the converted image set into an image classification network for training processing to obtain a backend network; Connecting the front-end network and the back-end network to obtain an overall identification network; The overall recognition network is converted into an accelerated reasoning framework, and is respectively deployed on a portable device and a cloud server to perform sign language recognition processing on the sign language action to be recognized, thereby obtaining a sign language recognition result; The front-end network includes a posture estimation network and a hand node recognition network. The front-end network is used to screen and crop the training image set to obtain a conversion image set, including: Performing key point detection processing on each training image in the training image set through the posture estimation network to obtain upper body posture nodes and node scores; The training image is screened according to the node score, and the screened image is intercepted to obtain a hand rectangular frame; Performing node recognition processing on the hand rectangular frame through the hand node recognition network to obtain the hand joint positions; Performing conversion processing on the training image according to the upper body posture nodes and the hand joint positions to obtain a conversion image set; The converting process of the training image according to the upper body posture nodes and the hand joint positions to obtain a converted image set includes: Get the black background image; According to the upper body posture nodes and the hand joint positions, corresponding positions in the black background image are illuminated and connected to obtain a skeleton image; The boundary position of the skeleton image is intercepted to obtain a conversion image set.
2. The method according to claim 1, It is characterized in that The step of inputting the converted image set into an image classification network for training to obtain a backend network comprises: Initializing the weight parameters of the image classification network to obtain an initial network; The converted image set is normalized and horizontally randomly flipped to obtain a training set and a validation set; The model parameters of the initial network are updated according to the training set and the validation set to obtain a backend network.
3. The method according to claim 1, It is characterized in that The portable device and the cloud server are respectively deployed to perform sign language recognition processing on the sign language action to be recognized to obtain a sign language recognition result, including: Performing image acquisition processing on the sign language action to be recognized by the portable device to obtain a sign language image to be recognized; Establishing a connection channel between the portable device and the cloud server; Selecting an inference mode according to the delay of the connection channel to perform inference recognition processing on the sign language image to be recognized, and obtaining an inference recognition result; The inference recognition result is evaluated and processed in combination with the overall recognition network to obtain a sign language recognition result.
4. The method according to claim 3, It is characterized in that The step of evaluating the inference recognition result in combination with the overall recognition network to obtain a sign language recognition result includes: Selecting and recording the training image set to obtain a sign language action video; Inputting the sign language action video frame by frame into the overall recognition network for recognition processing, and performing accuracy calculation processing on the recognition results to obtain an accuracy standard; The inference recognition result is screened according to the accuracy standard, and judged in combination with the inference speed and frame rate display of the inference recognition result to obtain the sign language recognition result.
5. The method according to any one of claims 1 to 4, It is characterized in that The step of acquiring a training image set comprises: Acquire sign language vocabulary, and record a sign language action video according to the sign language vocabulary; The sign language action video is subjected to key frame interception processing to obtain a training image set.
6. A lightweight sign language recognition system, It is characterized in that The system comprises: The first module is used to obtain a training image set; The second module is used to screen and crop the training image set through a front-end network to obtain a conversion image set; The third module is used to input the converted image set into the image classification network for training processing to obtain a backend network; The fourth module is used to connect the front-end network and the back-end network to obtain an overall identification network; The fifth module is used to perform accelerated reasoning framework conversion processing on the overall recognition network, and respectively deploy it on the portable device and the cloud server to perform sign language recognition processing on the sign language action to be recognized, so as to obtain a sign language recognition result; The front-end network includes a posture estimation network and a hand node recognition network. The second module is used to screen and crop the training image set through the front-end network to obtain a conversion image set, including: Performing key point detection processing on each training image in the training image set through the posture estimation network to obtain upper body posture nodes and node scores; The training image is screened according to the node score, and the screened image is intercepted to obtain a hand rectangular frame; Performing node recognition processing on the hand rectangular frame through the hand node recognition network to obtain the hand joint positions; Performing conversion processing on the training image according to the upper body posture nodes and the hand joint positions to obtain a conversion image set; The converting process of the training image according to the upper body posture nodes and the hand joint positions to obtain a converted image set includes: Get the black background image; According to the upper body posture nodes and the hand joint positions, corresponding positions in the black background image are illuminated and connected to obtain a skeleton image; The boundary position of the skeleton image is intercepted to obtain a conversion image set.
7. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
