Double-flow multi-fusion continuous sign language recognition method and device based on frame enhancement

Abstract

The invention discloses a double-flow multi-fusion continuous sign language recognition method and device based on frame enhancement, and the method comprises the steps: obtaining a skeleton thermodynamic diagram according to an RGB image; performing feature extraction on the RGB image and the skeleton thermodynamic diagram through a frame enhancement module and a bidirectional long-short-term memory network to obtain a first extraction feature of an RGB image flow and a second extraction feature of a thermodynamic diagram flow; obtaining a first target weighted mode and a second target weighted mode according to the first extraction feature and the second extraction feature; and obtaining a target identification result according to the first target weighted mode and the second target weighted mode. The method can improve the accuracy of continuous sign language recognition, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement. Background Art

[0002] Sign Language Recognition (SLR) is a computer vision (CV) technology that aims to enable computers to understand and interpret sign language movements. Currently, there is a lack of corresponding sign language translation resources in sign language applications. With the development of deep learning, using deep learning to achieve human-computer sign language interaction has become a solution. Developing a network model that can correctly understand and recognize sign language is an indispensable part of it. Some methods use 2D convolutional neural networks to perform recognition based on a single RGB image. However, such solutions fail to fully capture the dynamic information of sign language. In order to be able to be applied in real scenes, continuous sign language movements need to be recognized. Some studies have tried to use 3D convolution and motion trajectory modules to capture dynamic changes, but it is still difficult to accurately focus on key movements and areas. In addition, most of the current continuous sign language recognition methods only use RGB data for modeling to understand sign language, but such methods are less robust to background changes. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement, in order to solve at least one of the problems in the prior art and improve the accuracy of continuous sign language recognition.

[0004] To achieve the above object, an embodiment of the present invention provides a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, the method comprising:

[0005] Obtain the bone thermal map based on the RGB image;

[0006] Performing feature extraction on the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0007] Acquire a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0008] A target recognition result is obtained according to the first target weighted mode and the second target weighted mode.

[0009] In some embodiments, obtaining a bone thermal map according to the RGB image comprises the following steps:

[0010] Performing posture estimation on the person in the RGB image to obtain skeleton data;

[0011] A thermal map mapping operation is performed on the bone data to obtain the bone thermal map.

[0012] In some embodiments, the step of extracting features from the RGB image and the bone heat map to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream comprises the following steps:

[0013] Inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted features;

[0014] The skeleton heat map is input into the heat map stream of the backbone network, and features are extracted from the heat map stream to obtain the second extracted features.

[0015] In some embodiments, inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted feature comprises the following steps:

[0016] In the RGB image stream, preprocessing the RGB image to obtain a first intermediate feature;

[0017] Inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature;

[0018] According to the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature;

[0019] The third intermediate feature is input into the bidirectional long short-term memory network, and the first extracted feature is output.

[0020] In some embodiments, the step of inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature, and obtaining the second intermediate feature comprises the following steps:

[0021] Inputting the first intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the first intermediate feature, and obtaining a first spatial guidance feature;

[0022] Inputting the first intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the first intermediate feature to obtain a first motion guidance feature;

[0023] The first spatial guidance feature, the first motion guidance feature, and the first intermediate feature are summed to obtain the second intermediate feature.

[0024] In some embodiments, the step of inputting the skeletal heat map into the heat map stream of the backbone network, performing feature extraction on the heat map stream, and obtaining the second extracted feature comprises the following steps:

[0025] In the heat map stream, preprocessing the bone heat map to obtain a fourth intermediate feature;

[0026] Inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature;

[0027] According to the fifth intermediate feature, the heat map stream is interacted with the RGB image stream to obtain a sixth intermediate feature;

[0028] The sixth intermediate feature is input into the bidirectional long short-term memory network, and the second extracted feature is output.

[0029] In some embodiments, the step of inputting the fourth intermediate feature into the frame enhancement module and performing frame enhancement processing on the fourth intermediate feature to obtain the fifth intermediate feature comprises the following steps:

[0030] Inputting the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the fourth intermediate feature, and obtaining a second spatial guidance feature;

[0031] Inputting the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the fourth intermediate feature to obtain a second motion guidance feature;

[0032] The second spatial guiding feature, the second motion guiding feature and the fourth intermediate feature are summed to obtain the fifth intermediate feature.

[0033] In some embodiments, acquiring a first target weighted mode and a second target weighted mode according to the first extracted feature and the second extracted feature comprises the following steps:

[0034] Performing a concatenation operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature;

[0035] Performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted mode;

[0036] The first intermediate weighted mode is split to obtain the first target weighted mode and the second target weighted mode.

[0037] In some embodiments, obtaining a target recognition result according to the first target weighting mode and the second target weighting mode comprises the following steps:

[0038] Summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode;

[0039] The second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

[0040] To achieve the above object, another aspect of an embodiment of the present invention provides a dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, the device comprising:

[0041] The first module is used to obtain a bone thermal map based on the RGB image;

[0042] The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0043] A third module, configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0044] The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0047] To achieve the above-mentioned purpose, another aspect of the embodiment of the present invention provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the aforementioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0048] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement, and the scheme obtains a bone heat map based on an RGB image; performs feature extraction on the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; obtains a first target weighted modality and a second target weighted modality based on the first extracted feature and the second extracted feature; obtains a target recognition result based on the first target weighted modality and the second target weighted modality, which can improve the accuracy of continuous sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 is a flow chart of a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of a dual-stream feature extraction backbone network framework provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of a frame enhancement module provided by an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of a framework of an adaptive weighting module provided in an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0056] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0057] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0059] The study of sign language recognition, as a branch of deep learning in the field of computer vision, focuses on understanding and interpreting sign language visual signals. Early methods relied on hand-crafted features and hidden Markov models, but their accuracy was limited. With the advancement of technology, researchers began to use neural networks based on RGB images to extract features. However, existing methods ignore inter-frame motion when dealing with frame-level feature extraction. To solve this problem, some studies use 3D convolution or 2D+1D convolution variants to extract short-term temporal information. However, such methods are prone to overfitting on small data sets. Therefore, mainstream methods still rely on 2D CNN to extract frame-level features. Some existing sign language recognition methods use the 2DCNN+LSTM architecture for feature extraction, which is insufficient for modeling continuous sign language.

[0060] In addition, current sign language recognition models mainly use single-stream input of RGB images, but this type of modal information is very sensitive to problems such as lighting and environmental changes in continuous sign language. To improve recognition accuracy, researchers have introduced information from other modalities such as skeletons and depth maps, but existing methods usually lead to information loss or the final result is too dependent on the performance of a single modality due to information imbalance between different modalities.

[0061] In view of this, if Figure 1 As shown, the embodiment of the present invention provides a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, which may include but is not limited to steps S100 to S400:

[0062] Step S100, obtaining a bone thermal map according to the RGB image;

[0063] Step S200, performing feature extraction on the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0064] Step S300, acquiring a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0065] Step S400, obtaining a target recognition result according to the first target weighted mode and the second target weighted mode.

[0066] In some embodiments, in steps S100 to S400, skeleton information is introduced as a supplement to visual information to make up for the deficiency of the single modality of RGB images, making the model more robust to problems such as hand occlusion and changes in lighting conditions in sign language videos. And through the dual-stream architecture and adaptive weighting, the information between the two different modalities of skeleton information and RGB images is fully interacted and integrated, thereby improving the accuracy of continuous sign language recognition.

[0067] In some embodiments, step S100 may include but is not limited to steps S110 to S120:

[0068] Step S110, performing posture estimation on the person in the RGB image to obtain skeleton data;

[0069] Step S120, performing a heat map mapping operation on the skeleton data to obtain the skeleton heat map.

[0070] In step S110 of some embodiments, Figure 2 As shown in FIG, by estimating the posture of the person in the RGB image, the corresponding skeleton data of the human body can be obtained. For example, there is an RGB image of the tth frame in the video. in, H is the pixel height of the RGB image, and W is the pixel width of the RGB image. By performing posture estimation, we can obtain K skeletal joint points corresponding to the human body. in, is the pixel coordinate of the bone joint point in the RGB image, k = 1, 2, ..., K. By introducing bone data, the deficiency of the single modality of RGB images is made up.

[0071] In step S120 of some embodiments, the corresponding bone data is mapped into a heat map to obtain a bone heat map. For example, a bone heat map can be obtained by performing a heat map mapping operation on the bone data. Then the expression of the pixel value of the bone heat map is:

[0072]

[0073] In the formula, represents the pixel value at the kth channel (i, j) in the bone thermal map; i = 1, 2, ..., H; j = 1, 2, ..., W; σ represents the standard deviation; Represents the confidence score of the corresponding bone joint point obtained by pose estimation. By mapping the corresponding bone data to the heat map, the format of the RGB image and the bone data can be kept consistent.

[0074] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0075] Step S210, inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted features;

[0076] Step S220: input the skeleton heat map into the heat map stream of the backbone network, perform feature extraction on the heat map stream, and obtain the second extracted feature.

[0077] In steps S210 to S220 of some embodiments, Figure 2 As shown in the figure, the network structure used to extract RGB image features is the same as that of the bone heat map features. The feature extraction backbone network based on the dual-stream (RGB image stream and heat map stream) RGB image and bone data can extract features from the RGB image through the RGB image stream and extract features from the bone heat map through the heat map stream. By utilizing the features of both RGB image information and bone structure information, more accurate continuous sign language recognition can be achieved.

[0078] refer to Figure 2In the feature extraction backbone network based on dual-stream RGB images and skeleton data, each stream contains four residual layers and one 1D convolutional network, and then the extracted features of the two streams can be obtained through a bidirectional long short-term memory network. The RGB image and the skeleton heat map first pass through two residual layers. Before the subsequent two residual layers and 1D convolution, a frame enhancement module is set to achieve the network's self-attention to the key areas of space and enhance the network's ability to model the motion information of adjacent frames, thereby improving the frame-level feature extraction capability. Reference Figure 3 ,like Figure 3 The frame enhancement module framework shown in the figure includes two parts: the spatial guidance submodule and the motion guidance submodule. The initial input feature map first passes through the spatial guidance submodule. In the spatial guidance, different weights can be assigned to different positions in each frame image through self-attention calculation, thereby guiding the network to emphasize important areas. Let the initial input feature map be The self-attention calculation process is as follows:

[0079]

[0080]

[0081] Among them, Q t , K t and V t Respectively represent the query matrix, key matrix and value matrix; w q 、w k and w v They represent the corresponding weight matrices, which are learnable parameters; represents the initial input feature map of the t-th frame; softmax(·) represents the softmax function; represents the obtained spatially guided features.

[0082] Similarly, each frame of the initial input feature map first passes through the 2D convolution layer in the motion guidance submodule, and then passes through the forward difference operation. The process in the motion guidance submodule is shown in the following formula:

[0083]

[0084]

[0085]

[0086] in, Represents the intermediate input feature map of the tth frame output after the 2D convolution operation; Represents the initial input feature map of the t-1th frame; Represents the intermediate input feature map of the t-1th frame output after the 2D convolution operation; Conv2D(·) represents 2D convolution; Conv2D(·,1×1) represents convolution using a 1×1 convolution kernel; represents the features of each forward difference; cat(·) represents the concatenation operation; I F Represents splicing features; Represents the resulting motion-guided features.

[0087] Afterwards, and According to the corresponding weights and the initial input feature map of the tth frame By summing, we can get the output of frame enhancement As shown below:

[0088]

[0089] in, represents the output of frame enhancement; σ1 and σ2 are the corresponding weights.

[0090] like Figure 3 As shown, after frame enhancement processing, the information of the two streams is interacted through a bidirectional connection. The bidirectional connection process is shown in the following formula:

[0091]

[0092] in, and They represent the input of the heat map stream and RGB map stream in the bidirectional connection process respectively; and Respectively represent the output of the heat map stream and the RGB map stream in the bidirectional connection process; δ H and δ R are learnable parameters.

[0093] Then, after passing through the BiLSTM network, we can finally get the preliminary extracted features of the two streams, which are the first extracted features of the RGB image stream Second Extraction Feature of Heatmap Stream

[0094] In some embodiments, step S210 may include but is not limited to steps S211 to S214:

[0095] Step S211, in the RGB image stream, preprocessing the RGB image to obtain a first intermediate feature;

[0096] Step S212, inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature;

[0097] Step S213: according to the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature;

[0098] Step S214: input the third intermediate feature into the bidirectional long short-term memory network and output the first extracted feature.

[0099] In step S211 of some embodiments, in the RGB image stream of the dual-stream feature extraction backbone network, the RGB image is preprocessed through a residual layer to output a first intermediate feature. Figure 2 , the RGB image is preprocessed through two residual layers, and the first intermediate feature can be output. Optionally, convolution and normalization operations can be performed in the residual layer. After the convolution and normalization operations, the ReLU activation function can be applied to increase nonlinearity and help the network learn complex feature representations. The operations in the residual layer are not limited here.

[0100] In some embodiments, step S212 may include but is not limited to steps S2121 to S2123:

[0101] Step S2121, inputting the first intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the first intermediate feature, and obtaining a first spatial guidance feature;

[0102] Step S2122, inputting the first intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the first intermediate feature to obtain a first motion guidance feature;

[0103] Step S2123: sum the first spatial guidance feature, the first motion guidance feature, and the first intermediate feature to obtain the second intermediate feature.

[0104] In step S2121 of some embodiments, the first intermediate feature The motion guidance submodule of the input frame enhancement module performs self-attention calculation on the first intermediate feature to obtain the first spatial guidance feature The calculation process is as follows:

[0105]

[0106]

[0107]

[0108] in, and denote the first query matrix, the first key matrix and the first value matrix respectively; and are the weight matrix of the first query matrix, the weight matrix of the first key matrix, and the weight matrix of the first value matrix, all of which are learnable parameters; represents the first intermediate feature of the tth frame; represents the first spatial guiding feature.

[0109] In step S2122 of some embodiments, the first intermediate feature The motion guidance submodule of the input frame enhancement module performs convolution operation and forward difference operation on the first intermediate feature to obtain the first motion guidance feature The calculation process is as follows:

[0110]

[0111] in, Represents the intermediate input features of the tth frame output after the 2D convolution operation; Represents the first intermediate feature of the t-1th frame; Represents the intermediate input features of the t-1th frame output after the 2D convolution operation; I represents each first forward difference feature; F,R represents the first splicing feature; Represents the first motion guiding feature.

[0112] In step S2123 of some embodiments, the first spatial guidance feature First motion guidance feature and the first intermediate feature By summing, we can get the second intermediate feature The calculation formula is as follows:

[0113]

[0114] in, represents the second intermediate feature; σ 1,R and σ 2,R are the weight of the first spatial guidance feature and the weight of the first motion guidance feature respectively.

[0115] In steps S213 to S214 of some embodiments, in the RGB image stream, the second intermediate feature output after frame enhancement is passed through the residual layer or the output feature obtained after passing through the 1D convolutional network as the input of the heat map stream, so that the RGB image stream and the heat map stream interact, and after the interaction of the bidirectional connection, the RGB image stream outputs the third intermediate feature. Then the third intermediate feature is input into the bidirectional long short-term memory network (BiLSTM), and finally the first extracted feature of the RGB image stream can be obtained.

[0116] In some embodiments, step S220 may include but is not limited to steps S221 to S224:

[0117] Step S221, in the heat map stream, preprocessing the bone heat map to obtain a fourth intermediate feature;

[0118] Step S222, inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature;

[0119] Step S223, according to the fifth intermediate feature, interacting the heat map stream with the RGB image stream to obtain a sixth intermediate feature;

[0120] Step S224: input the sixth intermediate feature into the bidirectional long short-term memory network, and output the second extracted feature.

[0121] In step S221 of some embodiments, in the heat map stream of the dual-stream feature extraction backbone network, the skeleton heat map is preprocessed through a residual layer to output a fourth intermediate feature. Figure 2 , the bone heat map is preprocessed through two residual layers, and the fourth intermediate feature can be output. Optionally, convolution and normalization operations can be performed in the residual layer. After the convolution and normalization operations, the ReLU activation function can be applied to increase nonlinearity and help the network learn complex feature representations. The operations in the residual layer are not limited here.

[0122] In some embodiments, step S222 may include but is not limited to steps S2221 to S2223:

[0123] Step S2221, inputting the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the fourth intermediate feature, and obtaining a second spatial guidance feature;

[0124] Step S2222, inputting the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the fourth intermediate feature to obtain a second motion guidance feature;

[0125] Step S2223, summing the second spatial guidance feature, the second motion guidance feature and the fourth intermediate feature to obtain the fifth intermediate feature.

[0126] In step S2221 of some embodiments, the fourth intermediate feature The motion guidance submodule of the input frame enhancement module performs self-attention calculation on the fourth intermediate feature to obtain the first spatial guidance feature The calculation process is as follows:

[0127]

[0128]

[0129] in, and denote the second query matrix, the second key matrix, and the second value matrix respectively; and are the weight matrix of the second query matrix, the weight matrix of the second key matrix, and the weight matrix of the second value matrix, all of which are learnable parameters; represents the fourth intermediate feature of the t-th frame; Represents the second spatial guidance feature.

[0130] In step S2222 of some embodiments, the fourth intermediate feature The motion guidance submodule of the input frame enhancement module performs convolution operation and forward difference operation on the fourth intermediate feature to obtain the second motion guidance feature The calculation process is as follows:

[0131]

[0132] in, Represents the intermediate input features of the tth frame output after the 2D convolution operation; represents the fourth intermediate feature of the t-1th frame; Represents the intermediate input features of the t-1th frame output after the 2D convolution operation; I represents each second forward differential feature; F,H represents the second splicing feature; Represents the second motion guiding feature.

[0133] In step S2223 of some embodiments, the second spatial guidance feature Second motion guidance feature And the fourth intermediate feature By summing, we can get the fifth intermediate feature The calculation formula is as follows:

[0134]

[0135] in, represents the fifth intermediate characteristic; σ 1,H and σ 2,H are the weight of the second spatial guidance feature and the weight of the second motion guidance feature, respectively.

[0136] In steps S223 to S224 of some embodiments, in the heat map stream, the fifth intermediate feature output after frame enhancement is passed through the residual layer or the output feature obtained after passing through the 1D convolutional network as the input of the RGB image stream, so that the heat map stream and the RGB image stream are interacted, and after the interaction of the bidirectional connection, the heat map stream outputs the sixth intermediate feature. Then the sixth intermediate feature is input into the bidirectional long short-term memory network (BiLSTM), and finally the second extracted feature of the heat map stream can be obtained.

[0137] In steps S300 to S400 of some embodiments, Figure 4 As shown in the figure, the features extracted by the dual-stream backbone network are concatenated, and then the weighted modality is obtained through self-attention calculation, which is then split to obtain the weighted modality corresponding to RGB and bones. Finally, the weighted two modality features are directly summed and passed through a linear layer to obtain the final classification result. By setting an adaptive weighting module, the different importance levels between different modalities are emphasized to achieve effective late fusion.

[0138] In some embodiments, step S300 may include but is not limited to steps S310 to S330:

[0139] Step S310, performing a splicing operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature;

[0140] Step S320, performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted mode;

[0141] Step S330: split the first intermediate weighted mode to obtain the first target weighted mode and the second target weighted mode.

[0142] In steps S310 to S330 of some embodiments, the first feature extraction And the second extracted feature By performing the splicing operation, we can get the seventh intermediate feature The seventh intermediate feature By performing self-attention calculation, we can get the first intermediate weighted mode Exemplarily, the process of calculating self-attention for the seventh intermediate feature is as follows:

[0143]

[0144]

[0145] in, Indicates the seventh intermediate characteristic; represents the first intermediate weighted mode.

[0146] Then for the first intermediate weighted mode By splitting, we can get the first target weighted mode and the second objective weighted mode

[0147] In some embodiments, step S400 may include but is not limited to steps S410 to S420:

[0148] Step S410, summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode;

[0149] Step S420: input the second intermediate weighted mode into a linear layer to obtain the target recognition result.

[0150] In some embodiments, in step S410 to step S420, the first target weighting mode and the second objective weighted mode By summing up, a second intermediate weighted mode can be obtained, and then the second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

[0151] The combination of the adaptive weighted modules in the embodiments of the present invention allows joint learning of deep associations between modalities and promotes feature interaction, and then the fusion enhances the independence and robustness of each modality in the decision-making stage, thereby obtaining more accurate recognition results.

[0152] The embodiment of the present invention introduces skeleton information to make up for the shortcomings of the single mode of RGB images, making the model more robust to problems such as hand occlusion and changes in lighting conditions in sign language videos. In addition, the embodiment of the present invention proposes a dual-stream architecture including a bidirectional connection and an adaptive weighting module to ensure full interaction and fusion of information between the two different modes of skeleton information and RGB images. At the same time, by introducing a frame enhancement module and using motion guidance, the model can focus on the difference between the previous and next frames, making up for the shortcomings of the 2DCNN+LSTM architecture in short-term time modeling, and through spatial guidance, the model can focus on the key areas in each frame, improving the accuracy and robustness of the model.

[0153] The embodiment of the present invention further provides a dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, which can implement the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement. The device includes:

[0154] The first module is used to obtain a bone thermal map based on the RGB image;

[0155] The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0156] A third module, configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0157] The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] refer to Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0162] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0163] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement in an embodiment of the present invention;

[0164] Input / output interface 503, used to implement information input and output;

[0165] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0166] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0167] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0168] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0169] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0171] A dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement in an embodiment of the present invention has the following advantages:

[0172] 1. The embodiment of the present invention proposes a dual-stream multi-feature fusion network architecture, which can utilize the features of two modalities, RGB image information and bone structure information, and improve the accuracy of continuous sign language recognition by fusing the two modalities of RGB image and human skeleton.

[0173] 2. The embodiment of the present invention proposes a frame enhancement module including space guidance and motion guidance. Through the dual guidance of space and motion, the network can adaptively focus on the key areas of each frame and the frames with significant differences between the previous and the next frames.

[0174] 3. The embodiment of the present invention proposes a human skeleton feature represented in the form of a heat map, which can reduce the complexity of the model and maintain a consistent format with the RGB image.

[0175] 4. An embodiment of the present invention proposes a bidirectional connection module that can integrate intermediate representations from video and heat map branches to generate optimized modal features.

[0176] 5. The embodiment of the present invention proposes an adaptive weighting module, which can achieve effective late fusion by emphasizing the importance of different modules.

[0177] 6. Compared with other continuous sign language recognition methods, the embodiment of the present invention adopts bone structure information as a supplement, which can improve the robustness of network recognition; through a variety of fusion strategies, it can achieve better fusion of image and bone information; adopts a frame enhancement module, and improves the extraction effect of timing information by designing and combining two different guidance methods.

[0178] In summary, the embodiment of the present invention designs a dual-stream network architecture with RGB images and bone heat maps as inputs, while introducing bone information, ensuring data consistency, and using bidirectional connections to strengthen information interaction between different modalities. At the end of the model, an adaptive weighting module is used to adaptively focus on circles of different modalities, achieving more efficient and full fusion and improving recognition accuracy.

[0179] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0180] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0181] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0182] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0183] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0184] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0185] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0186] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0187] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, characterized in that: The following steps are involved: Obtain the bone thermal map based on the RGB image; Performing feature extraction on the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; Acquire a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features; A target recognition result is obtained according to the first target weighted mode and the second target weighted mode.

2. According to the frame enhancement-based dual-stream multi-fusion continuous sign language recognition method of claim 1, it is characterized in that: The step of obtaining a bone thermal map according to the RGB image includes the following steps: Performing posture estimation on the person in the RGB image to obtain skeleton data; A thermal map mapping operation is performed on the bone data to obtain the bone thermal map.

3. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 1, characterized in that: The step of extracting features from the RGB image and the bone heat map to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream comprises the following steps: Inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted features; The skeleton heat map is input into the heat map stream of the backbone network, and features are extracted from the heat map stream to obtain the second extracted features.

4. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 3 is characterized in that: The step of inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted feature comprises the following steps: In the RGB image stream, preprocessing the RGB image to obtain a first intermediate feature; Inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature; According to the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature; The third intermediate feature is input into the bidirectional long short-term memory network, and the first extracted feature is output.

5. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 4, characterized in that: The step of inputting the first intermediate feature into the frame enhancement module and performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature comprises the following steps: Inputting the first intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the first intermediate feature, and obtaining a first spatial guidance feature; Inputting the first intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the first intermediate feature to obtain a first motion guidance feature; The first spatial guidance feature, the first motion guidance feature, and the first intermediate feature are summed to obtain the second intermediate feature.

6. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 3, characterized in that: The step of inputting the skeleton heat map into the heat map stream of the backbone network, performing feature extraction on the heat map stream, and obtaining the second extracted feature comprises the following steps: In the heat map stream, preprocessing the bone heat map to obtain a fourth intermediate feature; Inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature; According to the fifth intermediate feature, the heat map stream and the RGB image stream are interacted to obtain a sixth intermediate feature; The sixth intermediate feature is input into the bidirectional long short-term memory network, and the second extracted feature is output.

7. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 6, characterized in that: The step of inputting the fourth intermediate feature into the frame enhancement module and performing frame enhancement processing on the fourth intermediate feature to obtain the fifth intermediate feature comprises the following steps: Inputting the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the fourth intermediate feature, and obtaining a second spatial guidance feature; Inputting the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the fourth intermediate feature to obtain a second motion guidance feature; The second spatial guiding feature, the second motion guiding feature and the fourth intermediate feature are summed to obtain the fifth intermediate feature.

8. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 1, characterized in that: The step of acquiring a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features comprises the following steps: Performing a concatenation operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature; Performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted mode; The first intermediate weighted mode is split to obtain the first target weighted mode and the second target weighted mode.

9. The method for dual-stream multi-fusion continuous sign language recognition based on frame enhancement according to claim 1, characterized in that: The step of obtaining a target recognition result according to the first target weighted mode and the second target weighted mode comprises the following steps: Summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode; The second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

10. A dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, characterized in that: include: The first module is used to obtain a bone thermal map based on the RGB image; The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; A third module, configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features; The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

Images