Sign language recognition method, device and electronic device based on information sharing mechanism

Abstract

The present invention discloses a sign language recognition method, device and electronic device based on an information sharing mechanism, including: obtaining a sign language video to be recognized; extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized; using the joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data as model input data, and inputting the model input data into an information sharing network configured with an information sharing mechanism for feature prediction, and obtaining feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data; and obtaining a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction results. The embodiments of the present invention can mine the nonlinear relationship between different skeleton features, thereby improving the feature representation capability and the accuracy of sign language action recognition, and can be widely used in the field of image recognition technology.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a sign language recognition method, device and electronic device based on an information sharing mechanism. Background Art

[0002] Sign language recognition is a hot topic in the field of computer vision. It can effectively solve the communication problem between the general public and hearing-impaired patients, and has been widely used in abnormal behavior monitoring of human-machine / robot interaction and medical assistance. Since skeleton data has good robustness, it has attracted many scholars to study skeleton-based sign language recognition. However, most of the current skeleton-based sign language recognition methods have the problems of insufficient utilization of input features, ineffective use of the relationship between features, insufficient feature extraction of network models, and no further use of the model's own information. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a sign language recognition method, device and electronic device based on an information sharing mechanism, which can mine the nonlinear relationship between different skeleton features, thereby improving the feature representation capability and the accuracy of sign language action recognition.

[0004] An aspect of an embodiment of the present invention provides a sign language recognition method based on an information sharing mechanism, including:

[0005] Get the sign language video to be recognized;

[0006] Extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be recognized;

[0007] The joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data;

[0008] A sign language recognition result corresponding to the sign language video to be recognized is obtained according to the feature prediction result.

[0009] Optionally, the extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be recognized includes:

[0010] Determining a target sign language action in the sign language video to be recognized;

[0011] Extract joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data corresponding to the target sign language action in the sign language video to be recognized.

[0012] Optionally, the extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified includes:

[0013] Extracting joint feature data and bone feature data corresponding to a target sign language action in the sign language video to be recognized;

[0014] generating joint motion feature data associated with the joint feature data according to the joint feature data, and using the joint feature data and the joint motion feature data as joint comprehensive feature data;

[0015] Generating bone motion feature data associated with the bone feature data according to the bone feature data, and using the bone feature data and the bone motion feature data as bone comprehensive feature data;

[0016] According to the joint comprehensive feature data and the bone comprehensive feature data, the angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified is determined; wherein the angle comprehensive feature data includes angle feature data and angle motion feature data.

[0017] Optionally, the information sharing network further includes a feature extraction network; the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, and feature prediction results corresponding to the model input data are obtained, including:

[0018] Inputting the joint comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the joint comprehensive feature data through a feature extraction network in the information sharing network, and obtaining a joint feature prediction result;

[0019] Inputting the bone comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the bone comprehensive feature data through a feature extraction network in the information sharing network to obtain a bone feature prediction result;

[0020] The angle comprehensive feature data is input into an information sharing network equipped with an information sharing mechanism, and feature prediction is performed on the angle comprehensive feature data through a feature extraction network in the information sharing network to obtain an angle feature prediction result.

[0021] Optionally, the information sharing mechanism in the information sharing network is to set an inter-feature sharing block and an inter-layer sharing block in the feature extraction network, and the feature extraction network in the information sharing network is used to perform feature prediction on the joint comprehensive feature data to obtain a joint feature prediction result, including:

[0022] Inputting the joint comprehensive feature data into a feature extraction network in the information sharing network;

[0023] The joint comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain a joint feature prediction result.

[0024] Optionally, performing feature prediction on the comprehensive bone feature data through a feature extraction network in the information sharing network to obtain a bone feature prediction result includes:

[0025] Inputting the skeleton comprehensive feature data into the feature extraction network in the information sharing network;

[0026] The skeleton comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain a skeleton feature prediction result.

[0027] Optionally, performing feature prediction on the angle comprehensive feature data through a feature extraction network in the information sharing network to obtain an angle feature prediction result includes:

[0028] Inputting the angle comprehensive feature data into a feature extraction network in the information sharing network;

[0029] The angle comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain an angle feature prediction result.

[0030] Optionally, obtaining a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result includes:

[0031] A weighted sum calculation is performed on the joint feature prediction results, the bone feature prediction results, and the angle feature prediction results to obtain a sign language recognition result corresponding to the sign language video to be recognized.

[0032] Another aspect of an embodiment of the present invention provides a sign language recognition device based on an information sharing mechanism, comprising:

[0033] A video acquisition module is used to acquire the sign language video to be recognized;

[0034] A feature extraction module, used to extract joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be identified;

[0035] A feature prediction module, used to use the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data as model input data, and input the model input data into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data;

[0036] The sign language recognition result acquisition module is used to obtain the sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result.

[0037] An embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0038] The memory is used to store programs;

[0039] The processor executes the program to implement the method described in the embodiment of the present invention.

[0040] In an embodiment of the present invention, first, a sign language video to be recognized is obtained; then, joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized are extracted; then, the joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein, the information sharing mechanism is used to realize data sharing between different feature data; finally, a sign language recognition result corresponding to the sign language video to be recognized is obtained according to the feature prediction results. The embodiment of the present invention can provide necessary input data for subsequent feature extraction and sign language recognition by acquiring a sign language video to be recognized. Then, by extracting joint comprehensive feature data, skeleton comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized, key features in sign language actions can be captured from different angles to establish a rich feature representation, wherein the key features are skeleton features corresponding to local features, joint features corresponding to global features and angle features corresponding to scale invariant features, which are helpful to more accurately recognize the meaning of sign language. In addition, an information sharing mechanism is introduced into the information sharing network, which can realize data sharing between different feature data, so that data of different features can complement each other and provide more comprehensive sign language information, thereby improving the accuracy of sign language recognition. Furthermore, the nonlinear relationship between different skeleton features can be mined through the information sharing mechanism, thereby improving the feature representation capability and the accuracy of sign language action recognition. Finally, a more accurate sign language recognition result corresponding to the sign language video to be recognized can be obtained according to the feature prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flowchart of the steps of a sign language recognition method based on an information sharing mechanism provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of generating skeleton motion feature data provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of the structure of an information sharing network provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the structure of a shared block between features provided by an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of the structure of an inter-layer shared block provided by an embodiment of the present invention;

[0047] Figure 6 is a structural block diagram of a sign language recognition device based on an information sharing mechanism provided by an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0049] Figure 8 It is a structural block diagram of a computer system provided by an embodiment of the present invention and suitable for implementing an electronic device of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] Currently, sign language recognition is a hot topic in the field of computer vision. It can effectively solve the communication problem between the general public and hearing-impaired patients, and has been widely used in abnormal behavior monitoring and medical assistance in human-machine / robot interaction. Since skeleton data has good robustness, it has attracted many scholars to study skeleton-based sign language recognition. However, most of the current skeleton-based sign language recognition methods have the problems of insufficient utilization of input features, ineffective utilization of the relationship between features, insufficient feature extraction of network models, and no further utilization of the model's own information.

[0052] In this regard, in an embodiment of the present invention, a sign language video to be recognized is first obtained; then, joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized are extracted; then, the joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein, the information sharing mechanism is used to realize data sharing between different feature data; finally, a sign language recognition result corresponding to the sign language video to be recognized is obtained according to the feature prediction results. The embodiment of the present invention can provide necessary input data for subsequent feature extraction and sign language recognition by acquiring a sign language video to be recognized. Then, by extracting joint comprehensive feature data, skeleton comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized, key features in sign language actions can be captured from different angles to establish a rich feature representation, wherein the key features are skeleton features corresponding to local features, joint features corresponding to global features and angle features corresponding to scale invariant features, which are helpful to more accurately recognize the meaning of sign language. In addition, an information sharing mechanism is introduced into the information sharing network, which can realize data sharing between different feature data, so that data of different features can complement each other and provide more comprehensive sign language information, thereby improving the accuracy of sign language recognition. Furthermore, the nonlinear relationship between different skeleton features can be mined through the information sharing mechanism, thereby improving the feature representation capability and the accuracy of sign language action recognition. Finally, a more accurate sign language recognition result corresponding to the sign language video to be recognized can be obtained according to the feature prediction result.

[0053] Reference Figure 1 , which shows a flow chart of the steps of a sign language recognition method based on an information sharing mechanism provided by an embodiment of the present invention; Figure 1 As shown, the method comprises the following steps:

[0054] S101, obtaining a sign language video to be recognized;

[0055] Among them, the sign language video is a sign language video to be identified, and the sign language video usually contains sign language movements. The sign language movements involve the movements of the hands and arms, and there are usually changes in the movement of the joints and the movement of the bones, that is, the changes in the joint coordinates and the changes in the length of the bones are involved in the sign language movements. Among them, the joint refers to the part where two or more bones are connected together in the human skeletal system. The joint can make the bones move relatively, and the movement of the body is achieved through the contraction and relaxation of the muscles. In the sign language movements, the joint can refer to various joints of the hand, such as wrist joints, finger joints, etc. In addition, the skeleton is a supporting structure composed of bones in the human body, which provides shape and stability for the body and provides support for the muscles, so that the human body can perform various movements. In the sign language movements, the skeleton can refer to the skeletal structure of the hand, including wrist bones, metacarpal bones, phalanges, etc. In the embodiment of the present invention, 27 nodes including the nose, hand and arm are used.

[0056] The video usually exists in the form of a video stream, and to some extent, the video can be called a video stream. It should be noted that in actual applications, video and video stream are two different concepts. For the convenience of explanation, in the embodiments of the present invention, no distinction is made when referring to video and video stream.

[0057] In the embodiment of the present invention, by acquiring the sign language video to be recognized, basic input data can be provided for subsequent feature extraction and processing.

[0058] S102, extracting joint comprehensive feature data, bone comprehensive feature data, and angle comprehensive feature data from the sign language video to be recognized;

[0059] In an optional embodiment, step S102 may include: determining a target sign language action in a sign language video to be identified; extracting joint comprehensive feature data, bone comprehensive feature data, and angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified.

[0060] Among them, the target sign language movement is the sign language movement that needs to be recognized in the sign language video to be recognized. When recognizing sign language, it is necessary to first determine the target sign language movement in the video, and then perform feature extraction and recognition on it. The target sign language movement can be a single gesture or a complete expression composed of a series of continuous gestures. It can be understood that for the selection and determination of the target sign language movement, technical personnel in this field can make a selection according to actual conditions, and the embodiments of the present invention do not limit this.

[0061] In the SLR (Sign Language Recognition) task, features that can effectively represent sign language movements need to be input into the model to recognize sign language movements. In the embodiment of the present invention, in order to input more abundant information, three pairs of feature data, namely, joint feature data and joint motion feature data representing global features, bone feature data and bone motion feature data representing local features, and angle feature data and angle motion feature data with scale invariance, are used for processing to fuse different features to obtain more comprehensive features, thereby improving the accuracy of sign language recognition.

[0062] Optionally, in some specific embodiments, extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data corresponding to a target sign language action in a sign language video to be identified may include: extracting joint feature data and bone feature data corresponding to a target sign language action in a sign language video to be identified; generating joint motion feature data associated with the joint feature data based on the joint feature data, and using the joint feature data and the joint motion feature data as joint comprehensive feature data; generating bone motion feature data associated with the bone feature data based on the bone feature data, and using the bone feature data and the bone motion feature data as bone comprehensive feature data; determining the angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified based on the joint comprehensive feature data and the bone comprehensive feature data; wherein the angle comprehensive feature data includes angle feature data and angle motion feature data.

[0063] For the comprehensive joint feature data, it may include joint feature data and joint motion feature data, wherein the joint feature data mainly refers to the joint, the joint is a feature containing global information, the joint is the coordinate information of the hand joint, which can represent the position information of the hand in space and reflect the overall movement of the hand, that is, it can effectively represent the change of the joint position during the execution of sign language actions; the joint motion feature data refers to the movement of the joint in space, that is, it can represent the joint coordinates in time change, which describes the position, direction and angle of the joint that changes with the passage of time, and the joint movement can also be expressed by rotation, bending, extension and other methods of the joint; wherein, the use of joint motion feature data can be understood as directly inputting time information into the network (model), providing the network with richer information.

[0064] For the comprehensive bone feature data, it may include bone feature data and bone motion feature data. Among them, the bone feature data mainly refers to the bones. The bone feature contains local feature information, is the most natural representation of the human body, and can effectively represent the connection and interaction between joints. In addition, the bone feature can represent the change in the relative distance between the parts of the body during the execution of sign language, and has rotation invariance (that is, the coordinate change caused by the rotation of the whole skeleton will not affect the bone feature); the bone motion feature data can refer to the movement of the bones in space, which describes the position, direction and angle of the bones that change over time. The bone movement can be realized by the combination of joint movements to express the sign language action; similarly, the use of bone motion feature data can also be understood as directly inputting time information into the network (model), providing the network with richer information.

[0065] Among them, the joint feature data can be obtained according to the joint coordinate position in the positioning target sign language action, such as fingers, wrists, elbows, etc.; then the angles, distances and other parameters between the joints can be calculated according to the joint information to describe the relative position and posture between the joints to obtain the joint motion feature data. The bone feature data can be obtained by obtaining the bone length, angle and other data to describe the overall structure of the hand, and then the bone motion feature data, such as bone displacement, rotation, etc., can be extracted according to the bone feature data to describe the overall movement of the hand.

[0066] Specifically, the joint feature data, the bone feature data and their corresponding motion feature data can be obtained by the following formula:

[0067] JM i =J i t+1 -J i t ,(i=1,2...27,t=1,2...T)

[0068] B i =J i -J k ,(i,k∈Q)

[0069] BM i =B i t+1 -B i t ,(t=1,2...T)

[0070] Wherein, J represents the joint coordinates, JM represents the joint motion, i represents the nth joint (in the embodiment of the present invention, 27 joint nodes including the nose, hand and arm are used), B represents the bone, JM represents the i Represented as the joint motion feature of the i-th joint, B iRepresented as the i-th bone, BM i Represents the skeletal motion characteristics of the i-th joint.

[0071] Reference Figure 2 , showing a schematic diagram of generating skeleton motion feature data provided by an embodiment of the present invention; Figure 2 As shown, first, several skeletons (such as Figure 2 1, 2, and 3) and several joints (such as Figure 2 Joint 1, Joint 2 and Joint 3), because the bones and joints are related when they move, there are changes in joint positions and bone positions. Figure 2 In the figure, bones 1, 2, and 3 can form a coherent action. Figure 2 As shown, bones 1 and bones 2 can form bone movement 1, bones 2 and bones 3 can form bone movement 2, wherein the action combined by bone movement 1 and bone movement 2 is coherent, similarly, joints 1 and joints 2 can form joint movement 1, joints 2 and joints 3 can form joint movement 2, wherein the action combined by joint movement 1 and joint movement 2 is also coherent. It can be understood that in the recognition of sign language actions, the features of joints, joint movements, bones and bone movements are not isolated, but are closely related. Joints are a feature containing global information, which can effectively represent the changes in joint positions during the execution of sign language actions. Skeletal features contain local feature information, which can effectively represent the connection and interaction between joints. Movement features can effectively describe the frequency of action changes and can distinguish the amplitude of similar action changes rather than the frequency. These feature data can complement each other and can more comprehensively describe the features of sign language actions and improve the accuracy of sign language recognition.

[0072] For angle comprehensive feature data, it can include angle feature data and angle motion feature data. In the process of skeleton data collection, joint coordinates and bone length will change with the change of the distance between the sign language performer and the camera, wherein the change of joint coordinates and the change of bone length caused by the different camera distances will be regarded as the noise of the network (model), thereby affecting the recognition of the network, and the angle feature data is introduced in the embodiment of the present invention, and the angle feature data has scale invariance, wherein the angle feature data can be obtained according to the natural connection of the human skeleton to obtain the angle feature data between the skeletons, that is, the angle feature data can be understood as the angle existing in the state of the joint or skeleton in the natural connection, when performing joint movement or skeleton movement, even if the distances shot by different cameras are different, the angle feature data and angle motion feature data between the skeletons in the sign language action shot by the camera are also unchanged, so it can overcome the problem that the change of joint coordinates and the change of bone length caused by the different camera distances are regarded as the noise of the network, thereby affecting the recognition of the network.

[0073] It should be pointed out that after obtaining the skeleton features (i.e., joint feature data, bone feature data, and angle feature data) at different distances from the camera, these features can be visualized to obtain the corresponding feature map. Among them, since the joint coordinates and bone lengths will change with the change of the distance between the sign language performer and the camera, similarly, the visualized joint feature map and bone feature map will change according to the change of the distance between the sign language performer and the camera, while the angle feature map remains unchanged due to the scale invariance of the angle feature data.

[0074] In addition, the angular motion characteristic data can be obtained by the following formula:

[0075]

[0076]

[0077] Among them, A represents the angle, i and j represent the two joints used for calculation, S is the middle point (assuming that an angle has three points, then S is the middle point, for example: the natural connection lines between the two shoulders and the nose form an angle, then the nose is the middle point), t is the time frame, P is the pre-defined middle point, N is the set of all points, and Am is the angle movement.

[0078] In an embodiment of the present invention, after determining the target sign language action in the sign language video to be identified, extract the joint feature data and the bone feature data corresponding to the target sign language action in the sign language video to be identified; generate the joint motion feature data associated with the joint feature data according to the joint feature data, and use the joint feature data and the joint motion feature data as joint comprehensive feature data; generate the bone motion feature data associated with the bone feature data according to the bone feature data, and use the bone feature data and the bone motion feature data as bone comprehensive feature data; determine the angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified according to the joint comprehensive feature data and the bone comprehensive feature data; wherein the angle comprehensive feature data includes angle feature data and angle motion feature data. By extracting the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data in the sign language video to be identified, the key features in the sign language action can be captured from different angles to establish a rich feature representation, wherein the key features are the bone features corresponding to the local features, the joint features corresponding to the global features, and the angle features corresponding to the scale invariant features, so that the data of different features can complement each other, provide more comprehensive sign language information, and thus improve the accuracy of sign language recognition.

[0079] S103, using the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data as model input data, and inputting the model input data into an information sharing network configured with an information sharing mechanism for feature prediction, to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data;

[0080] Among them, the model input data includes joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data, specifically the aforementioned joint feature data and joint motion feature data, bone feature data and bone motion feature data, and angle feature data and angle motion feature data. The model input data can be feature extracted (predicted) through an information sharing network to obtain feature prediction results corresponding to the model input data.

[0081] Optionally, the information sharing network also includes a feature extraction network, which can be called a multi-stream feature extraction structure or a multi-stream end-to-end network, wherein multi-stream means that multiple feature data can be input into the feature extraction network to achieve feature extraction of multiple features, thereby improving the efficiency of feature prediction.

[0082] At present, if a single artificially weighted input feature is used for sign language prediction, an end-to-end model cannot be realized, in which early features correspond to high-level semantic features and are direct representations of actions, but if these features are rigidly connected in series, the different information contained in each feature may be blurred, thereby affecting the comprehensive extraction ability of the network. In an embodiment of the present invention, the model design is intended to input multiple features into the network at the same time, and then generate the final recognition result. In order to meet the demand for an end-to-end model with multiple input features, an embodiment of the present invention constructs a multi-stream end-to-end network (i.e., the aforementioned feature extraction network), which can be used to effectively extract each feature through multiple feature extraction branches, and finally weight the prediction results of each feature stream to obtain the final prediction result. In view of the large number of sign language categories, over-emphasizing the difference in output category probabilities may lead to model overfitting. In an embodiment of the present invention, a label smoothed cross entropy loss function is used to improve this problem, and the calculation formula of the total label smoothed cross entropy loss function of the model is as follows:

[0083] Loss=loss1+loss2+loss3+loss4+loss5+loss6

[0084] Among them, Loss is the total label smoothed cross entropy loss function of the model, which is obtained by weighted sum of each tributary label smoothed cross entropy loss function (such as loss1, loss2, loss3, loss4, loss5, loss6).

[0085] Reference Figure 3 , showing a schematic diagram of the structure of an information sharing network provided by an embodiment of the present invention; Figure 3 As shown, Figure 3 The GCN in the figure is the feature extraction layer of the feature extraction network. The feature extraction layer (FE layer for short) is the most critical layer in each feature extraction flow. Each branch consists of ten FE layers, and its purpose is to explore the potential information in the input features to facilitate the modeling of sign language actions. It can be understood that the feature extraction network consists of six feature extraction branches (such as Figure 3 There are six feature extraction branches corresponding to the six feature streams of joints, joint motion, bones, bone motion, angles, and angle motion in the figure), each of which consists of ten FE layers ( Figure 3 The six feature extraction branches can be used as inputs for six corresponding feature streams (such as Figure 3It should be noted that the number of feature extraction branches of the feature extraction network and the number of FE layers constructed by each branch can be adjusted by technicians in this field according to actual conditions, and the embodiment of the present invention does not limit this. In order to deal with the intricate interactions between hands and arms in sign language recognition tasks, it is necessary to aggregate spatial information from different body parts for different sign language movements. In order to meet this requirement, the embodiment of the present invention adopts decoupled GCN to construct the FE layer. The method of using decoupled GCN to construct the FE layer in the embodiment of the present invention integrates different adjacency matrices in different channels, so that GCN can aggregate node information according to their respective connection patterns. In addition, the decoupled GCN incorporates the Graph Drop mechanism by introducing additional regularization terms. Unlike dropout in the field of image processing, Drop Graph deletes the node itself and its neighbors, which helps to alleviate the overfitting of the model to a certain extent. The decoupled GCN is obtained by decoupling the GCN (Graph Convolutional Network). It should be noted that for how to decouple the GCN to obtain the decoupled GCN, those skilled in the art can refer to the relevant technology, and the embodiments of the present invention will not be described in detail here. Specifically, the calculation process and decoupling mechanism of the decoupled GCN can be expressed by the following formula:

[0086]

[0087]

[0088] When g equals 1, the decoupled GCN is essentially the same as the regular GCN, where is the input of the model, g is the number of groups in the decoupling, Both W are divided into g groups and multiplied with g different adjacency matrices to represent the connection operation. Drop refers to the probability of discarding the region, which is determined by the retention probability (Prob keep ) and the number of dropped nodes (Drop size ) to determine.

[0089] It should be pointed out that in the embodiment of the present invention, in order to ensure that the network pays attention to important areas, the decoupled GCN structure also adopts a spatiotemporal channel attention mechanism (STC-attention, referred to as STC attention), which allows the network to filter out redundant information and focus on key areas in space, time and channel dimensions.

[0090] In an optional embodiment, step S103 may include: inputting the joint comprehensive feature data into an information sharing network configured with an information sharing mechanism, performing feature prediction on the joint comprehensive feature data through a feature extraction network in the information sharing network, and obtaining a joint feature prediction result; inputting the bone comprehensive feature data into an information sharing network configured with an information sharing mechanism, performing feature prediction on the bone comprehensive feature data through a feature extraction network in the information sharing network, and obtaining a bone feature prediction result; inputting the angle comprehensive feature data into an information sharing network configured with an information sharing mechanism, performing feature prediction on the angle comprehensive feature data through a feature extraction network in the information sharing network, and obtaining an angle feature prediction result.

[0091] Among them, in various deep learning tasks such as image classification, object recognition and posture estimation, multiple features are usually used to achieve excellent results. However, related technologies usually process each input feature independently during the calculation process, and then directly perform prediction or feature fusion at the end of the calculation. The feature extraction network provided in the embodiment of the present invention is a multi-stream network architecture, which can ensure that valuable information is effectively extracted from each input feature, and a multi-level information sharing mechanism is set on this multi-stream network architecture to ensure that the complementary information between different features at the feature level is effectively utilized, thereby improving the utilization rate of information between features and cross-layer information. In addition, the problem of information loss during forward propagation can also be solved.

[0092] Optionally, the information sharing mechanism in the information sharing network is to set inter-feature sharing blocks and inter-layer sharing blocks in the feature extraction network. It can be understood that the information sharing network is a feature extraction network configured with an information sharing mechanism. Multiple features can be input into the information sharing network for feature prediction to obtain sign language recognition results. Therefore, the information sharing network can be called an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network (three multi sharingnetwork referred to as TMS-Net). Figure 3 As shown, Figure 3 The circles in represent the shared blocks between features, and the small rectangles (without text content) represent the shared blocks between layers. For example, Figure 3 The joint feature data and joint motion feature data in the feature extraction process can be shared through the inter-feature sharing block, the bone feature data and the bone motion feature data can be shared through the inter-feature sharing block, and the angle feature data and the angle motion feature data can be shared through the inter-feature sharing block. The complementary information between different features can be effectively utilized through the inter-feature sharing block; in addition, in order to solve the problem of some information loss in the forward propagation process during the feature extraction process, the inter-layer sharing block can be used to promote the sharing of different layers (different layers refer to Figure 3The information sharing between the ten FE layers (which perform feature extraction at different stages) in each feature extraction branch can combine and enhance the valuable information between different layers to obtain a more comprehensive representation.

[0093] Specifically, the final prediction result of the model is a combination of multiple feature prediction results. In the forward reasoning process, it is crucial to couple these orthogonal and non-orthogonal features together to construct a more comprehensive feature representation, rather than simply adding them together. To this end, the embodiment of the present invention designs a feature sharing block to promote information sharing between features. In the sign language recognition task, the model will gradually extract the input features and form different threshold areas, which are manifested as brightness changes in different areas of the feature map. Among them, the highlighted areas in the feature map can be interpreted as the positive impact of body part features on sign language action recognition. Conversely, the darker areas in the map can be interpreted as areas that need to be suppressed or inhibited. The feature sharing block uses this feature to integrate information from different features to achieve information sharing between multiple features. Reference Figure 4 , which shows a schematic diagram of the structure of a feature-to-feature sharing block provided by an embodiment of the present invention, such as Figure 4 As shown, exemplarily, the overall processing process of the shared block between features is: first, different skeleton features f {1,2...k} ∈R B×C×T×N (like Figure 4 The features 2...n) in the image are subjected to a normalization process Norm() to align the data distribution of different skeleton features and ensure a consistent representation, where B, C, T, and N represent the batch size, number of channels, time dimension, and number of points, respectively. Then, the normalized features are concatenated together to form a new feature block F∈R B×k*C×T×N (Feature block F before pooling), then, in the absolute value maximum pooling layer APool() designed with shared blocks between features, the excitation and inhibition regions in each feature are effectively retained. These regions contain the key information of the feature. This design is different from the traditional maximum pooling layer. B×C×T×N After (pooling feature block F), for this feature F∈R B×C×T×N and features that need to be shared Perform a dot product operation ⊙, and generate a new information matrix M∈R after the dot product operation B×C×T×N , the new information matrix M captures the interaction information between multiple features and shared feature information. In addition, the embodiment of the present invention introduces two learnable parameters β {1,2} ( Figure 4 β1 and β2 in , used to identify this new information matrix M and the features to be shared Weighted (can be understood as a connection operation ). It should be noted that this example is only a brief schematic illustration of the general process of sharing blocks between features. The parameters mentioned in this example can allow the network to dynamically adjust and optimize the contribution of different information sources to the recognition task during the training process. It can be understood that those skilled in the art can adjust the relevant parameters according to actual conditions, and the embodiments of the present invention are not limited to this.

[0094] Specifically, during the feature extraction process, features will gradually transform from higher-level semantic features to lower-level detail features. Therefore, this gradual transformation will lead to some information loss in the forward propagation process (i.e., early feature information is more valuable when performing recognition tasks). To solve this problem, the embodiments of the present invention provide inter-layer sharing blocks to promote information sharing between different layers, thereby combining and enhancing valuable information between different layers to obtain a more comprehensive representation. Unlike methods that process different features, inter-layer features belong to the same type of features, so no normalization operation is required. Figure 5 , shows a schematic diagram of the structure of an inter-layer shared block provided by an embodiment of the present invention, such as Figure 5 As shown in Figure 1, the overall processing process of the inter-layer sharing block is as follows: First, the inter-layer sharing block uses a size matching operation to ensure that the input features from different layers have matching sizes. If the sizes are inconsistent, operations such as pooling are applied to align them. Then, the features of the previous layer are input into the STC attention module ( Figure 5 Finally, we introduce two learnable parameters ( Figure 5 a1 and a2 in the figure add the features of the current layer to the processed features of the previous layer (which can be understood as a connection operation). ), a comprehensive feature containing high-level semantic information and fine-grained detail information is obtained, which is then used for forward propagation. Where ⊙ represents the dot product operation, Indicates a connection operation. It should be noted that this example is only a brief schematic illustration of the general process of inter-layer sharing blocks. The parameters mentioned in this example can allow the network to dynamically adjust and optimize the contribution of different information sources to the recognition task during training. It is understandable that those skilled in the art can adjust the relevant parameters according to actual conditions, and the embodiments of the present invention are not limited to this.

[0095] Optionally, in some specific embodiments, the joint comprehensive feature data is input into the feature extraction network in the information sharing network, and the joint comprehensive feature data can be predicted through the feature sharing blocks and inter-layer sharing blocks in the feature extraction network to obtain joint feature prediction results. Similarly, the bone comprehensive feature data is input into the feature extraction network in the information sharing network, and the bone comprehensive feature data can be predicted through the feature sharing blocks and inter-layer sharing blocks in the feature extraction network to obtain bone feature prediction results. Similarly, the angle comprehensive feature data is input into the feature extraction network in the information sharing network, and the angle comprehensive feature data can be predicted through the feature sharing blocks and inter-layer sharing blocks in the feature extraction network to obtain angle feature prediction results.

[0096] Among them, the joint feature prediction results include joint prediction results and joint movement prediction results, the joint prediction results correspond to the joint feature data in the joint comprehensive feature data, and the joint movement prediction results correspond to the joint movement feature data in the joint comprehensive feature data; the bone feature prediction results include bone prediction results and bone movement prediction results, the bone prediction results correspond to the bone feature data in the bone comprehensive feature data, and the bone movement prediction results correspond to the bone movement feature data in the bone comprehensive feature data; the angle feature prediction results include angle prediction results and angle movement prediction results, the angle prediction results correspond to the angle feature data in the angle comprehensive feature data, and the angle movement prediction results correspond to the angle movement feature data in the angle comprehensive feature data.

[0097] In an embodiment of the present invention, three pairs of feature data, namely, joint feature data and joint motion feature data, bone feature data and bone motion feature data, and angle feature data and angle motion feature data, are input into an information sharing network configured with an information sharing mechanism. The joint feature data and joint motion feature data can be predicted by the feature sharing block and the inter-layer sharing block in the feature extraction network to obtain the joint feature prediction result. Similarly, the bone feature data and bone motion feature data can be predicted by the feature sharing block and the inter-layer sharing block in the feature extraction network to obtain the bone feature prediction result. Similarly, the bone feature data and bone motion feature data can be predicted by the feature sharing block and the inter-layer sharing block in the feature extraction network to obtain the angle feature prediction result. By using the feature extraction network in the information sharing network to extract features from multiple feature data, the complementary information between different features is effectively utilized, and the results such as joint features, bone features and angle features can be predicted more accurately, thereby improving the accuracy of motion posture estimation. Moreover, by using the feature sharing block and the inter-layer sharing block in the feature extraction network, different feature data can be shared and learned, thereby better extracting more valuable information from various features.

[0098] S104. Obtain a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result.

[0099] In an optional embodiment, step S104 may include: performing weighted sum calculation on the joint feature prediction results, the bone feature prediction results, and the angle feature prediction results to obtain a sign language recognition result corresponding to the sign language video to be recognized.

[0100] In the specific implementation, it is necessary to perform a weighted sum calculation on the joint feature prediction results, bone feature prediction results and angle feature prediction results obtained in the information sharing network (that is, perform a weighted sum calculation on the joint prediction results and joint movement prediction results, the bone prediction results and the bone movement prediction results, and the angle prediction results and the angle movement prediction results), and use the final prediction result obtained by the weighted sum operation as the sign language recognition result corresponding to the sign language video to be recognized.

[0101] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0102] like Figure 3 As shown, the embodiment of the present invention provides an end-to-end skeleton-based multi-feature multi-stream multi-level information sharing network, the information sharing network is composed of multiple input features, a multi-stream feature extraction network and a multi-level information sharing mechanism, that is, multiple input features can be input into the multi-stream feature extraction network, and feature prediction is performed on multiple features through the multi-stream feature extraction network and the multi-level information sharing mechanism set therein, thereby obtaining a sign language prediction result. Figure 3 As shown, the feature extraction network (multi-stream feature extraction network) consists of six feature extraction branches, each of which consists of ten FE layers ( Figure 3 The six feature extraction branches can be used as inputs for six corresponding feature streams (such as Figure 3 The six feature streams of joints, joint motions, bones, bone motions, angles, and angular motions in the model are used for feature detection. The input feature data can exchange information through shared blocks between features to achieve interaction between different features. In addition, inter-layer shared blocks are used to combine and enhance valuable information between different layers to obtain a more comprehensive representation. The final prediction result of the model is obtained by combining the prediction results of each data stream through a weighted sum operation.

[0103] Specifically, first, the sign language video to be recognized is obtained and the input features in the target sign language action are extracted, the input features including joint feature data and joint motion feature data corresponding to the global features, bone feature data and bone motion feature data corresponding to the local features, and angle feature data and angle motion feature data corresponding to the scale-invariant features. First, these three pairs of feature data are respectively input into each feature extraction branch in the multi-stream feature extraction network. Then, the joint feature data and the joint motion feature data can be shared through the inter-feature sharing block, the bone feature data and the bone motion feature data can be shared through the inter-feature sharing block, and the angle feature data and the angle motion feature data can be shared through the inter-feature sharing block. At the same time, the inter-layer sharing block can be used to promote the sharing of different layers (different layers can refer to Figure 3 The information is shared between the ten FE layers (which perform feature extraction at different stages) in each feature extraction branch, and then the joint prediction results corresponding to the joint feature data, the joint movement prediction results corresponding to the joint movement feature data, the bone prediction results corresponding to the bone feature data, the bone movement prediction results corresponding to the bone movement feature data, the angle prediction results corresponding to the angle feature data, and the angle movement prediction results corresponding to the angle movement feature data are obtained. Finally, the joint prediction results, the joint movement prediction results, the bone prediction results, the bone movement prediction results, the angle prediction results, and the angle movement prediction results are weighted summed, and the result obtained by the calculation is used as the sign language recognition result in the sign language video to be recognized. It should be pointed out that this embodiment only briefly schematically illustrates the general process of the sign language recognition method based on the information sharing mechanism. The detailed description of each step can refer to the relevant content in the aforementioned embodiment, and will not be repeated here. It can be understood that the present invention is not limited to this.

[0104] In an embodiment of the present invention, first, a sign language video to be recognized is obtained; then, joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized are extracted; then, the joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein, the information sharing mechanism is used to realize data sharing between different feature data; finally, a sign language recognition result corresponding to the sign language video to be recognized is obtained according to the feature prediction results. The embodiment of the present invention can provide necessary input data for subsequent feature extraction and sign language recognition by acquiring a sign language video to be recognized. Then, by extracting joint comprehensive feature data, skeleton comprehensive feature data and angle comprehensive feature data in the sign language video to be recognized, key features in sign language actions can be captured from different angles to establish a rich feature representation, wherein the key features are skeleton features corresponding to local features, joint features corresponding to global features and angle features corresponding to scale invariant features, which are helpful to more accurately recognize the meaning of sign language. In addition, an information sharing mechanism is introduced into the information sharing network, which can realize data sharing between different feature data, so that data of different features can complement or merge each other, provide more comprehensive sign language information, and thus improve the accuracy of sign language recognition. Furthermore, the nonlinear relationship between different skeleton features can be mined through the information sharing mechanism, thereby improving the feature representation capability and the accuracy of sign language action recognition. Finally, a more accurate sign language recognition result corresponding to the sign language video to be recognized can be obtained according to the feature prediction result.

[0105] In addition, the feature extraction network in the information sharing network can efficiently extract multiple features of the input, thereby improving the prediction efficiency of the network. The information sharing mechanism can realize the process of data sharing between different feature data. Specifically, the complementary information between different features can be effectively utilized through the inter-feature sharing blocks in the information sharing mechanism, and the inter-layer sharing blocks in the information sharing mechanism can promote information sharing between different layers, thereby effectively solving the problem of some information loss in the forward propagation process during the feature extraction process.

[0106] The embodiment of the present invention also provides a sign language recognition device based on an information sharing mechanism, referring to Figure 6 , shows a structural block diagram of a sign language recognition device based on an information sharing mechanism provided by an embodiment of the present invention, the device includes the following modules:

[0107] The video acquisition module 601 is used to acquire the sign language video to be recognized;

[0108] A feature extraction module 602 is used to extract joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be recognized;

[0109] The feature prediction module 603 is used to use the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data as model input data, and input the model input data into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data;

[0110] The sign language recognition result acquisition module 604 is used to obtain the sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result.

[0111] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0112] On the other hand, Figure 7 As shown, an embodiment of the present invention further provides an electronic device 700, which includes at least one processor 710 and at least one memory 720 for storing at least one program; taking one processor 710 and one memory 720 as an example.

[0113] The processor 710 and the memory 720 may be connected via a bus or in other ways.

[0114] The memory 720, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] Specifically, Figure 8The structure block diagram of a computer system of an electronic device for implementing an embodiment of the present invention is schematically shown.

[0117] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0118] like Figure 8 As shown, the computer system 800 includes a central processing unit 801 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 802 (ROM) or the program loaded from the storage part 808 to the random access memory 803 (RAM). Various programs and data required for system operation are also stored in the random access memory 803. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output interface 805 (Input / Output interface, i.e., I / O interface) is also connected to the bus 804.

[0119] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0120] In particular, according to an embodiment of the present invention, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processor 801, various functions defined in the system of the present invention are executed.

[0121] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0122] The contents of the method embodiments of the present invention are all applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method.

[0123] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0124] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0128] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0129] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0130] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0131] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0133] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0134] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0135] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0137] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A sign language recognition method based on information sharing mechanism, characterized in that: include: Get the sign language video to be recognized; Extracting joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be recognized; The joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data; Obtaining a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result; Wherein, the joint comprehensive feature data includes joint feature data and joint motion feature data, the bone comprehensive feature data includes bone feature data and bone motion feature data, the angle comprehensive feature data includes angle feature data and angle motion feature data, the information sharing network includes a feature extraction network, and the information sharing mechanism in the information sharing network is to set an inter-feature sharing block and an inter-layer sharing block in the feature extraction network, the inter-feature sharing block is used to share information between the joint feature data and the joint motion feature data, the inter-feature sharing block is used to share information between the bone feature data and the bone motion feature data, the inter-feature sharing block is used to share information between the angle feature data and the angle motion feature data, and the inter-layer sharing block is used to share information between different feature extraction layers in the same feature extraction branch; The joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data are used as model input data, and the model input data is input into an information sharing network configured with an information sharing mechanism for feature prediction, to obtain feature prediction results corresponding to the model input data, including: Inputting the joint comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the joint comprehensive feature data through a feature extraction network in the information sharing network, and obtaining a joint feature prediction result; Inputting the bone comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the bone comprehensive feature data through a feature extraction network in the information sharing network to obtain a bone feature prediction result; The angle comprehensive feature data is input into an information sharing network equipped with an information sharing mechanism, and feature prediction is performed on the angle comprehensive feature data through a feature extraction network in the information sharing network to obtain an angle feature prediction result.

2. The method according to claim 1, characterized in that The step of extracting joint comprehensive feature data, bone comprehensive feature data, and angle comprehensive feature data from the sign language video to be recognized includes: Determining a target sign language action in the sign language video to be recognized; Extract joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data corresponding to the target sign language action in the sign language video to be recognized.

3. The method according to claim 2, characterized in that The step of extracting joint comprehensive feature data, bone comprehensive feature data, and angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified includes: Extracting joint feature data and bone feature data corresponding to a target sign language action in the sign language video to be recognized; generating joint motion feature data associated with the joint feature data according to the joint feature data, and using the joint feature data and the joint motion feature data as joint comprehensive feature data; Generating bone motion feature data associated with the bone feature data according to the bone feature data, and using the bone feature data and the bone motion feature data as bone comprehensive feature data; According to the joint comprehensive feature data and the bone comprehensive feature data, angle comprehensive feature data corresponding to the target sign language action in the sign language video to be identified is determined.

4. The method according to claim 1, characterized in that: The step of performing feature prediction on the joint comprehensive feature data through the feature extraction network in the information sharing network to obtain a joint feature prediction result includes: Inputting the joint comprehensive feature data into a feature extraction network in the information sharing network; The joint comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain a joint feature prediction result.

5. The method according to claim 1, characterized in that The step of performing feature prediction on the comprehensive bone feature data through the feature extraction network in the information sharing network to obtain a bone feature prediction result includes: Inputting the skeleton comprehensive feature data into the feature extraction network in the information sharing network; The skeleton comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain a skeleton feature prediction result.

6. The method according to claim 1, characterized in that The step of performing feature prediction on the angle comprehensive feature data through the feature extraction network in the information sharing network to obtain an angle feature prediction result includes: Inputting the angle comprehensive feature data into a feature extraction network in the information sharing network; The angle comprehensive feature data is predicted through the inter-feature sharing block and the inter-layer sharing block in the feature extraction network to obtain an angle feature prediction result.

7. The method according to claim 1, characterized in that The step of obtaining a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result includes: A weighted sum calculation is performed on the joint feature prediction results, the bone feature prediction results, and the angle feature prediction results to obtain a sign language recognition result corresponding to the sign language video to be recognized.

8. A sign language recognition device based on an information sharing mechanism, characterized in that: include: A video acquisition module is used to acquire the sign language video to be recognized; A feature extraction module, used to extract joint comprehensive feature data, bone comprehensive feature data and angle comprehensive feature data from the sign language video to be identified; A feature prediction module, used to use the joint comprehensive feature data, the bone comprehensive feature data and the angle comprehensive feature data as model input data, and input the model input data into an information sharing network configured with an information sharing mechanism for feature prediction, so as to obtain feature prediction results corresponding to the model input data; wherein the information sharing mechanism is used to realize data sharing between different feature data; A sign language recognition result acquisition module, used to obtain a sign language recognition result corresponding to the sign language video to be recognized according to the feature prediction result; Wherein, the joint comprehensive feature data includes joint feature data and joint motion feature data, the bone comprehensive feature data includes bone feature data and bone motion feature data, the angle comprehensive feature data includes angle feature data and angle motion feature data, the information sharing network includes a feature extraction network, and the information sharing mechanism in the information sharing network is to set an inter-feature sharing block and an inter-layer sharing block in the feature extraction network, the inter-feature sharing block is used to share information between the joint feature data and the joint motion feature data, the inter-feature sharing block is used to share information between the bone feature data and the bone motion feature data, the inter-feature sharing block is used to share information between the angle feature data and the angle motion feature data, and the inter-layer sharing block is used to share information between different feature extraction layers in the same feature extraction branch; The feature prediction module is specifically used for: Inputting the joint comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the joint comprehensive feature data through a feature extraction network in the information sharing network, and obtaining a joint feature prediction result; Inputting the bone comprehensive feature data into an information sharing network equipped with an information sharing mechanism, performing feature prediction on the bone comprehensive feature data through a feature extraction network in the information sharing network to obtain a bone feature prediction result; The angle comprehensive feature data is input into an information sharing network equipped with an information sharing mechanism, and feature prediction is performed on the angle comprehensive feature data through a feature extraction network in the information sharing network to obtain an angle feature prediction result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

Images