A dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement

Abstract

The present invention discloses a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement. The method comprises: obtaining a skeletal heat map based on an RGB image; performing feature extraction on the RGB image and the skeletal heat map using a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; obtaining a first target weighted modality and a second target weighted modality based on the first extracted feature and the second extracted feature; and obtaining a target recognition result based on the first target weighted modality and the second target weighted modality. The present invention can improve the accuracy of continuous sign language recognition and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement. Background Art

[0002] Sign language recognition (SLR) is a computer vision (CV) technology designed to enable computers to understand and interpret sign language movements. Currently, sign language applications lack corresponding sign language translation resources. With the development of deep learning, leveraging deep learning to achieve human-computer sign language interaction has become a solution. Developing network models that can correctly understand and recognize sign language is essential. Some methods utilize two-dimensional convolutional neural networks for recognition based on a single RGB image. However, these approaches fail to fully capture the dynamic nature of sign language. For real-world application, continuous sign language movement recognition is necessary. Some studies have attempted to capture dynamic changes using 3D convolutional and motion trajectory modules, but they still struggle to accurately focus on key movements and areas. Furthermore, most current continuous sign language recognition methods use only RGB data for modeling to understand sign language, but these methods are less robust to background variations. Summary of the Invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement, in order to solve at least one of the problems in the existing technology and improve the accuracy of continuous sign language recognition.

[0004] To achieve the above objectives, an embodiment of the present invention provides a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, the method comprising:

[0005] Obtain bone heat map based on RGB image;

[0006] Performing feature extraction on the RGB image and the bone heat map using a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0007] Acquire a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0008] A target recognition result is obtained according to the first target weighted mode and the second target weighted mode.

[0009] In some embodiments, obtaining a bone thermal map based on an RGB image comprises the following steps:

[0010] Performing posture estimation on the person in the RGB image to obtain skeleton data;

[0011] A heat map mapping operation is performed on the skeleton data to obtain the skeleton heat map.

[0012] In some embodiments, the step of extracting features from the RGB image and the bone heat map to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream includes the following steps:

[0013] Inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream to obtain the first extracted features;

[0014] The skeleton heat map is input into the heat map stream of the backbone network, and feature extraction is performed on the heat map stream to obtain the second extracted feature.

[0015] In some embodiments, inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted features comprises the following steps:

[0016] In the RGB image stream, preprocessing the RGB image to obtain a first intermediate feature;

[0017] Inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature;

[0018] According to the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature;

[0019] The third intermediate feature is input into the bidirectional long short-term memory network, and the first extracted feature is output.

[0020] In some embodiments, inputting the first intermediate feature into the frame enhancement module and performing frame enhancement processing on the first intermediate feature to obtain the second intermediate feature includes the following steps:

[0021] Inputting the first intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the first intermediate feature to obtain a first spatial guidance feature;

[0022] Inputting the first intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the first intermediate feature to obtain a first motion guidance feature;

[0023] The first spatial guidance feature, the first motion guidance feature, and the first intermediate feature are summed to obtain the second intermediate feature.

[0024] In some embodiments, inputting the skeleton heat map into the heat map stream of the backbone network, performing feature extraction on the heat map stream, and obtaining the second extracted features comprises the following steps:

[0025] In the heat map stream, preprocessing the bone heat map to obtain a fourth intermediate feature;

[0026] Inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature;

[0027] According to the fifth intermediate feature, interacting the heat map stream with the RGB image stream to obtain a sixth intermediate feature;

[0028] The sixth intermediate feature is input into the bidirectional long short-term memory network, and the second extracted feature is output.

[0029] In some embodiments, inputting the fourth intermediate feature into the frame enhancement module and performing frame enhancement processing on the fourth intermediate feature to obtain the fifth intermediate feature includes the following steps:

[0030] Inputting the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the fourth intermediate feature to obtain a second spatial guidance feature;

[0031] Inputting the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the fourth intermediate feature to obtain a second motion guidance feature;

[0032] The second spatial guidance feature, the second motion guidance feature, and the fourth intermediate feature are summed to obtain the fifth intermediate feature.

[0033] In some embodiments, obtaining a first target weighted mode and a second target weighted mode based on the first extracted features and the second extracted features includes the following steps:

[0034] performing a splicing operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature;

[0035] Performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted modality;

[0036] The first intermediate weighted mode is split to obtain the first target weighted mode and the second target weighted mode.

[0037] In some embodiments, obtaining a target recognition result according to the first target weighting mode and the second target weighting mode comprises the following steps:

[0038] Summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode;

[0039] The second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

[0040] To achieve the above objectives, another aspect of the present invention provides a dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, the device comprising:

[0041] The first module is used to obtain the bone heat map based on the RGB image;

[0042] The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0043] A third module is configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0044] The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the aforementioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0047] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0048] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement, which obtains a bone heat map based on an RGB image; performs feature extraction on the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; obtains a first target weighted modality and a second target weighted modality based on the first extracted feature and the second extracted feature; obtains a target recognition result based on the first target weighted modality and the second target weighted modality, which can improve the accuracy of continuous sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement provided by an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the dual-stream feature extraction backbone network framework provided by an embodiment of the present invention;

[0052] Figure 3 1 is a schematic diagram of a frame enhancement module provided by an embodiment of the present invention;

[0053] Figure 4 1 is a schematic diagram of a framework of an adaptive weighting module provided in an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0056] It should be noted that although the functional modules are divided in the system schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned figures may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination".

[0057] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0059] Sign language recognition research, a branch of deep learning in computer vision, focuses on understanding and interpreting sign language visual signals. Early methods relied on handcrafted features and hidden Markov models, but their accuracy was limited. With technological advancements, researchers began using neural networks based on RGB images to extract features. However, existing methods ignore inter-frame motion when extracting frame-level features. To address this issue, some studies have used 3D convolution or 2D+1D convolution variants to extract short-term temporal information. However, these methods are prone to overfitting on small datasets. Therefore, mainstream methods still rely on 2D CNNs to extract frame-level features. Some existing sign language recognition methods use a 2D CNN+LSTM architecture for feature extraction, which is insufficient for modeling continuous sign language.

[0060] Furthermore, current sign language recognition models primarily use a single stream of RGB images as input, but this modality is highly sensitive to issues such as lighting and environmental changes in continuous sign language. To improve recognition accuracy, researchers have introduced information from other modalities, such as skeletons and depth maps. However, existing methods often result in information loss or an imbalance in information between different modalities, making the final results overly dependent on the performance of a single modality.

[0061] In view of this, if Figure 1 As shown, the embodiment of the present invention provides a dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, which may include but is not limited to steps S100 to S400:

[0062] Step S100, obtaining a bone thermal map based on the RGB image;

[0063] Step S200: performing feature extraction on the RGB image and the bone heat map using a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0064] Step S300: Acquire a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0065] Step S400: Obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

[0066] In some embodiments, steps S100 through S400 incorporate skeletal information as a supplement to visual information, addressing the limitations of the single-modality RGB image. This makes the model more robust to issues such as hand occlusion and changing lighting conditions in sign language videos. Furthermore, through a dual-stream architecture and adaptive weighting, the interaction and fusion of skeletal information and RGB imagery improve the accuracy of continuous sign language recognition.

[0067] In some embodiments, step S100 may include but is not limited to steps S110 to S120:

[0068] Step S110, performing posture estimation on the person in the RGB image to obtain skeleton data;

[0069] Step S120 , performing a heat map mapping operation on the skeleton data to obtain the skeleton heat map.

[0070] In step S110 of some embodiments, Figure 2 As shown in the figure, by estimating the posture of the person in the RGB image, the corresponding skeleton data of the human body can be obtained. For example, there is an RGB image of the tth frame in the video. in, H is the pixel height of the RGB image, and W is the pixel width of the RGB image. Performing posture estimation, we can obtain K skeletal joint points corresponding to the human body in, is the pixel coordinate of the skeleton joint in the RGB image, k = 1, 2, ..., K. By introducing the skeleton data, the deficiency of the single modality of the RGB image is made up.

[0071] In step S120 of some embodiments, the corresponding bone data is mapped to the heat map to obtain a bone heat map. For example, the bone heat map can be obtained by performing a heat map mapping operation on the bone data. Then the expression of the pixel value of the bone heat map is:

[0072]

[0073] Where, represents the pixel value at the kth channel (i, j) in the bone heat map; i = 1, 2, ..., H; j = 1, 2, ..., W; σ represents the standard deviation; Represents the confidence score of the corresponding skeletal joint point obtained from the pose estimation. By mapping the corresponding skeletal data to the heat map, the format of the RGB image and the skeletal data can be kept consistent.

[0074] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0075] Step S210: input the RGB image into the RGB image stream of the backbone network, perform feature extraction on the RGB image stream, and obtain the first extracted features;

[0076] Step S220: input the skeleton heat map into the heat map stream of the backbone network, perform feature extraction on the heat map stream, and obtain the second extracted feature.

[0077] In steps S210 to S220 of some embodiments, Figure 2 As shown in the figure, the network structure used to extract RGB image features and bone heat map features is the same. The feature extraction backbone network based on the dual-stream (RGB image stream and heat map stream) RGB image and bone data can extract features from RGB images through the RGB image stream and extract features from bone heat maps through the heat map stream. By utilizing the features of both RGB image information and bone structure information, more accurate continuous sign language recognition can be achieved.

[0078] refer to Figure 2In the feature extraction backbone network based on dual-stream RGB images and skeleton data, each stream contains four residual layers and one 1D convolutional network. Then, the extracted features of the two streams can be obtained through a bidirectional long short-term memory network. The RGB image and the skeleton heat map first pass through two residual layers. Before the subsequent two residual layers and the 1D convolution, a frame enhancement module is set up to achieve the network's self-attention to key spatial areas and enhance the network's ability to model the motion information of adjacent frames, thereby improving the frame-level feature extraction capability. Figure 3 ,like Figure 3 The frame enhancement module framework shown in the figure consists of two parts: the spatial guidance submodule and the motion guidance submodule. The initial input feature map first passes through the spatial guidance submodule. In the spatial guidance, different weights can be assigned to different positions in each frame image through self-attention calculation, thereby guiding the network to emphasize important areas. Let the initial input feature map be The self-attention calculation process is as follows:

[0079]

[0080]

[0081] Among them, Q t , K t and V t Represent the query matrix, key matrix and value matrix respectively; w q 、w k and w v Represent the corresponding weight matrices, which are learnable parameters; represents the initial input feature map of the t-th frame; softmax(·) represents the softmax function; represents the obtained spatially guided features.

[0082] Similarly, each frame of the initial input feature map first passes through the 2D convolution layer in the motion guidance submodule, and then passes through the forward difference operation. The process in the motion guidance submodule is shown in the following formula:

[0083]

[0084]

[0085]

[0086] in, Represents the intermediate input feature map of the t-th frame output after the 2D convolution operation; represents the initial input feature map of the t-1th frame; Represents the intermediate input feature map of the t-1th frame output after the 2D convolution operation; Conv2D(·) represents 2D convolution; Conv2D(·,1×1) represents convolution using a 1×1 convolution kernel; represents the features of each forward difference; cat(·) represents the concatenation operation; I F Represents splicing features; Represents the resulting motion-guided features.

[0087] Afterwards, and According to the corresponding weight and the initial input feature map of the tth frame By summing, we can get the output of frame enhancement As shown in the following formula:

[0088]

[0089] in, represents the output of frame enhancement; σ1 and σ2 are the corresponding weights.

[0090] like Figure 3 As shown, after frame enhancement processing, the information of the two streams is exchanged through a bidirectional connection. The bidirectional connection process is shown in the following formula:

[0091]

[0092] in, and They represent the input of the heat map stream and RGB image stream in the bidirectional connection process respectively; and Respectively represent the output of the heat map stream and RGB map stream in the bidirectional connection process; δ H and δ R is a learnable parameter.

[0093] Then, after passing through the Bidirectional Long Short-Term Memory Network (BiLSTM), we can finally get the preliminary extracted features of the two streams, which are the first extracted features of the RGB image stream Second extracted feature of heat map flow

[0094] In some embodiments, step S210 may include but is not limited to steps S211 to S214:

[0095] Step S211: preprocessing the RGB image in the RGB image stream to obtain a first intermediate feature;

[0096] Step S212: inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature;

[0097] Step S213: Based on the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature;

[0098] Step S214: input the third intermediate feature into the bidirectional long short-term memory network and output the first extracted feature.

[0099] In step S211 of some embodiments, in the RGB image stream of the dual-stream feature extraction backbone network, the RGB image is preprocessed through the residual layer to output the first intermediate feature. Figure 2 , the RGB image is preprocessed through two residual layers to output the first intermediate features. Optionally, convolution and normalization operations can be performed in the residual layer. After convolution and normalization, the ReLU activation function can be applied to increase nonlinearity and help the network learn complex feature representations. The operations in the residual layer are not limited here.

[0100] In some embodiments, step S212 may include but is not limited to steps S2121 to S2123:

[0101] Step S2121: input the first intermediate feature into the spatial guidance submodule of the frame enhancement module, perform self-attention calculation on the first intermediate feature, and obtain a first spatial guidance feature;

[0102] Step S2122: input the first intermediate feature into the motion guidance submodule of the frame enhancement module, perform convolution and forward difference operations on the first intermediate feature, and obtain a first motion guidance feature;

[0103] Step S2123: sum the first spatial guidance feature, the first motion guidance feature, and the first intermediate feature to obtain the second intermediate feature.

[0104] In step S2121 of some embodiments, the first intermediate feature The motion guidance submodule of the input frame enhancement module performs self-attention calculation on the first intermediate feature to obtain the first spatial guidance feature The calculation process is as follows:

[0105]

[0106]

[0107]

[0108] in, and denote the first query matrix, the first key matrix, and the first value matrix respectively; and are the weight matrices of the first query matrix, the first key matrix, and the first value matrix, respectively, all of which are learnable parameters; represents the first intermediate feature of the t-th frame; represents the first spatial guidance feature.

[0109] In step S2122 of some embodiments, the first intermediate feature The motion guidance submodule of the input frame enhancement module performs convolution operation and forward difference operation on the first intermediate feature to obtain the first motion guidance feature The calculation process is as follows:

[0110]

[0111] in, Represents the intermediate input features of the t-th frame output after the 2D convolution operation; represents the first intermediate feature of the t-1th frame; Represents the intermediate input features of the t-1th frame output after the 2D convolution operation; Represents each first forward differential feature; I F,R represents the first splicing feature; Represents the first motion guidance feature.

[0112] In step S2123 of some embodiments, the first spatial guidance feature First motion guidance feature and the first intermediate feature By summing, we can get the second intermediate feature The calculation formula is as follows:

[0113]

[0114] in, represents the second intermediate feature; σ 1,R and σ 2,R are the weight of the first spatial guidance feature and the weight of the first motion guidance feature, respectively.

[0115] In some embodiments, in steps S213 to S214, in the RGB image stream, the second intermediate feature output after frame enhancement is passed through the residual layer or the output feature obtained after passing through the 1D convolutional network as the input of the heat map stream, so that the RGB image stream and the heat map stream are interacted. After the interaction of the bidirectional connection, the RGB image stream outputs the third intermediate feature. The third intermediate feature is then input into the bidirectional long short-term memory network (BiLSTM), and finally the first extracted feature of the RGB image stream can be obtained.

[0116] In some embodiments, step S220 may include but is not limited to steps S221 to S224:

[0117] Step S221: preprocessing the skeleton heat map in the heat map stream to obtain a fourth intermediate feature;

[0118] Step S222: inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature;

[0119] Step S223: Based on the fifth intermediate feature, the heat map stream and the RGB image stream are interacted to obtain a sixth intermediate feature;

[0120] Step S224: input the sixth intermediate feature into the bidirectional long short-term memory network and output the second extracted feature.

[0121] In step S221 of some embodiments, in the heat map stream of the dual-stream feature extraction backbone network, the skeleton heat map is preprocessed through the residual layer to output the fourth intermediate feature. Figure 2 The skeletal heatmap is preprocessed through two residual layers to output the fourth intermediate feature. Optionally, convolution and normalization operations can be performed in the residual layer. After convolution and normalization, the ReLU activation function can be applied to increase nonlinearity and help the network learn complex feature representations. The operations in the residual layer are not limited here.

[0122] In some embodiments, step S222 may include but is not limited to steps S2221 to S2223:

[0123] Step S2221: input the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, perform self-attention calculation on the fourth intermediate feature, and obtain a second spatial guidance feature;

[0124] Step S2222: input the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, perform convolution and forward difference operations on the fourth intermediate feature, and obtain a second motion guidance feature;

[0125] Step S2223: sum the second spatial guidance feature, the second motion guidance feature, and the fourth intermediate feature to obtain the fifth intermediate feature.

[0126] In step S2221 of some embodiments, the fourth intermediate feature The motion guidance submodule of the input frame enhancement module performs self-attention calculation on the fourth intermediate feature to obtain the first spatial guidance feature The calculation process is as follows:

[0127]

[0128]

[0129] in, and denote the second query matrix, the second key matrix, and the second value matrix respectively; and are the weight matrices of the second query matrix, the weight matrix of the second key matrix, and the weight matrix of the second value matrix, respectively, all of which are learnable parameters; represents the fourth intermediate feature of the t-th frame; Represents the second spatial guidance feature.

[0130] In step S2222 of some embodiments, the fourth intermediate feature The motion guidance submodule of the input frame enhancement module performs convolution operation and forward difference operation on the fourth intermediate feature to obtain the second motion guidance feature The calculation process is as follows:

[0131]

[0132] in, Represents the intermediate input features of the t-th frame output after the 2D convolution operation; represents the fourth intermediate feature of the t-1th frame; Represents the intermediate input features of the t-1th frame output after the 2D convolution operation; Represents each second forward differential feature; I F,H represents the second splicing feature; Represents the second motion guidance feature.

[0133] In step S2223 of some embodiments, the second spatial guidance feature Second motion guidance feature and the fourth intermediate feature By summing, we can get the fifth intermediate feature The calculation formula is as follows:

[0134]

[0135] in, represents the fifth intermediate characteristic; σ 1,H and σ 2,H are the weights of the second spatial guidance feature and the second motion guidance feature, respectively.

[0136] In some embodiments, in steps S223 to S224, in the heat map stream, the fifth intermediate feature output after frame enhancement is passed through the residual layer or the output feature obtained after passing through the 1D convolutional network as the input of the RGB image stream, so that the heat map stream and the RGB image stream interact with each other. After the interaction of the bidirectional connection, the heat map stream outputs the sixth intermediate feature. The sixth intermediate feature is then input into the bidirectional long short-term memory network (BiLSTM), and finally the second extracted feature of the heat map stream can be obtained.

[0137] In steps S300 to S400 of some embodiments, Figure 4 As shown, the features extracted by the two-stream backbone network are concatenated and then calculated using self-attention to obtain a weighted modality. This is then split to obtain the weighted modalities corresponding to RGB and skeleton. Finally, the weighted features of the two modalities are directly summed and passed through a linear layer to obtain the final classification result. By setting up an adaptive weighting module, the varying importance of different modalities is emphasized, enabling effective late-stage fusion.

[0138] In some embodiments, step S300 may include but is not limited to steps S310 to S330:

[0139] Step S310: performing a splicing operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature;

[0140] Step S320: performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted modality;

[0141] Step S330: split the first intermediate weighted mode to obtain the first target weighted mode and the second target weighted mode.

[0142] In steps S310 to S330 of some embodiments, the first feature extraction And the second extracted feature Perform the splicing operation to obtain the seventh intermediate feature The seventh intermediate feature Performing self-attention calculation, we can get the first intermediate weighted mode For example, the process of calculating self-attention for the seventh intermediate feature is as follows:

[0143]

[0144]

[0145] in, Indicates the seventh intermediate characteristic; represents the first intermediate weighted mode.

[0146] Then for the first intermediate weighted mode By splitting, we can get the first target weighted mode and the second objective weighted modality

[0147] In some embodiments, step S400 may include but is not limited to steps S410 to S420:

[0148] Step S410: summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode;

[0149] Step S420: input the second intermediate weighted mode into a linear layer to obtain the target recognition result.

[0150] In steps S410 to S420 of some embodiments, the first target weighting mode and the second objective weighted modality By summing up, a second intermediate weighted mode can be obtained, and then the second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

[0151] The combination of the adaptive weighting modules in the embodiments of the present invention allows for joint learning of deep associations between modalities and promotes feature interaction, and the subsequent fusion enhances the independence and robustness of each modality in the decision-making stage, thereby obtaining more accurate recognition results.

[0152] The embodiment of the present invention introduces skeletal information to compensate for the shortcomings of the single modality of RGB images, making the model more robust to problems such as hand occlusion and changes in lighting conditions in sign language videos. In addition, the embodiment of the present invention proposes a dual-stream architecture including bidirectional connections and an adaptive weighting module to ensure full interaction and fusion of information between the two different modalities of skeletal information and RGB images. At the same time, by introducing a frame enhancement module and adopting motion guidance, the model can focus on the differences between the previous and next frames, which compensates for the shortcomings of the 2DCNN+LSTM architecture in short-term time modeling, and through spatial guidance, the model can focus on the key areas in each frame, thereby improving the accuracy and robustness of the model.

[0153] An embodiment of the present invention further provides a dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, which can implement the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement. The device includes:

[0154] The first module is used to obtain the bone heat map based on the RGB image;

[0155] The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream;

[0156] A third module is configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features;

[0157] The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned method for continuous sign language recognition using dual-stream multi-fusion based on frame enhancement. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0160] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] refer to Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0162] The processor 501 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0163] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 502 and are called by the processor 501 to execute the dual-stream multi-fusion continuous sign language recognition method based on frame enhancement according to the embodiment of the present invention.

[0164] Input / output interface 503, used to implement information input and output;

[0165] Communication interface 504, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0166] Bus 505 , which transmits information between various components of the device (e.g., processor 501 , memory 502 , input / output interface 503 , and communication interface 504 );

[0167] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0168] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned dual-stream multi-fusion continuous sign language recognition method based on frame enhancement.

[0169] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] An embodiment of the present invention further provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned frame-enhanced dual-stream multi-fusion continuous sign language recognition method.

[0171] The dual-stream multi-fusion continuous sign language recognition method and device based on frame enhancement according to the embodiment of the present invention have the following advantages:

[0172] 1. The embodiment of the present invention proposes a dual-stream multi-feature fusion network architecture that can utilize the features of both RGB image information and bone structure information. By fusing the two modalities of RGB image and human skeleton, the accuracy of continuous sign language recognition is improved.

[0173] 2. An embodiment of the present invention proposes a frame enhancement module that includes spatial guidance and motion guidance. Through dual spatial and motion guidance, the network can adaptively focus on the key areas of each frame and frames with significant differences between the previous and next frames.

[0174] 3. The embodiment of the present invention proposes a human skeleton feature represented in the form of a heat map, which can reduce the complexity of the model and maintain a consistent format with the RGB image.

[0175] 4. An embodiment of the present invention proposes a bidirectional connection module that can integrate intermediate representations from video and heat map branches to generate optimized modal features.

[0176] 5. The embodiment of the present invention proposes an adaptive weighting module, which can achieve effective late fusion by emphasizing the importance of different modules.

[0177] 6. Compared with other continuous sign language recognition methods, the embodiment of the present invention uses skeletal structure information as a supplement to improve the robustness of network recognition; through multiple fusion strategies, it can achieve better fusion of image and skeletal information; and adopts a frame enhancement module to improve the extraction effect of temporal information by designing and combining two different guidance methods.

[0178] In summary, the present invention employs a dual-stream network architecture designed with RGB images and skeletal heatmaps as input. This architecture ensures data consistency while introducing skeletal information, and uses bidirectional connections to enhance information exchange between different modalities. Finally, the model employs an adaptive weighting module to adaptively focus on circles of different modalities, achieving more efficient and comprehensive fusion and improving recognition accuracy.

[0179] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0180] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0181] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0182] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0183] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0184] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0185] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0186] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0187] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A dual-stream multi-fusion continuous sign language recognition method based on frame enhancement, characterized in that: The following steps are involved: Obtain bone heat map based on RGB image; The RGB image and the bone heat map are subjected to feature extraction through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream. The frame enhancement module includes a spatial guidance submodule and a motion guidance submodule. The spatial guidance submodule obtains spatial guidance features through self-attention calculation. The motion guidance submodule obtains motion guidance features through a convolution layer and forward difference operation. After passing through the frame enhancement module, the RGB image stream and the heat map stream are interactively connected through a bidirectional connection. Acquire a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features; A target recognition result is obtained according to the first target weighted mode and the second target weighted mode.

2. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 1, characterized in that: The method of obtaining a bone thermal map based on the RGB image includes the following steps: Performing posture estimation on the person in the RGB image to obtain skeleton data; A heat map mapping operation is performed on the skeleton data to obtain the skeleton heat map.

3. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 1, characterized in that: The step of extracting features from the RGB image and the bone heat map to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream includes the following steps: Inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream to obtain the first extracted features; The skeleton heat map is input into the heat map stream of the backbone network, and feature extraction is performed on the heat map stream to obtain the second extracted feature.

4. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 3, characterized in that: Inputting the RGB image into the RGB image stream of the backbone network, performing feature extraction on the RGB image stream, and obtaining the first extracted features comprises the following steps: In the RGB image stream, preprocessing the RGB image to obtain a first intermediate feature; Inputting the first intermediate feature into the frame enhancement module, performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature; According to the second intermediate feature, the RGB image stream and the heat map stream are interacted to obtain a third intermediate feature; The third intermediate feature is input into the bidirectional long short-term memory network, and the first extracted feature is output.

5. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 4 is characterized in that: The step of inputting the first intermediate feature into the frame enhancement module and performing frame enhancement processing on the first intermediate feature to obtain a second intermediate feature comprises the following steps: Inputting the first intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the first intermediate feature to obtain a first spatial guidance feature; Inputting the first intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the first intermediate feature to obtain a first motion guidance feature; The first spatial guidance feature, the first motion guidance feature, and the first intermediate feature are summed to obtain the second intermediate feature.

6. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 3, characterized in that: Inputting the skeleton heat map into the heat map stream of the backbone network, performing feature extraction on the heat map stream, and obtaining the second extracted features, comprises the following steps: In the heat map stream, preprocessing the bone heat map to obtain a fourth intermediate feature; Inputting the fourth intermediate feature into the frame enhancement module, performing frame enhancement processing on the fourth intermediate feature to obtain a fifth intermediate feature; According to the fifth intermediate feature, interacting the heat map stream with the RGB image stream to obtain a sixth intermediate feature; The sixth intermediate feature is input into the bidirectional long short-term memory network, and the second extracted feature is output.

7. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 6, characterized in that: The step of inputting the fourth intermediate feature into the frame enhancement module and performing frame enhancement processing on the fourth intermediate feature to obtain the fifth intermediate feature comprises the following steps: Inputting the fourth intermediate feature into the spatial guidance submodule of the frame enhancement module, performing self-attention calculation on the fourth intermediate feature to obtain a second spatial guidance feature; Inputting the fourth intermediate feature into the motion guidance submodule of the frame enhancement module, performing a convolution operation and a forward difference operation on the fourth intermediate feature to obtain a second motion guidance feature; The second spatial guidance feature, the second motion guidance feature, and the fourth intermediate feature are summed to obtain the fifth intermediate feature.

8. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 1, characterized in that: The step of obtaining a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features comprises the following steps: performing a splicing operation on the first extracted feature and the second extracted feature to obtain a seventh intermediate feature; Performing self-attention calculation on the seventh intermediate feature to obtain a first intermediate weighted modality; The first intermediate weighted mode is split to obtain the first target weighted mode and the second target weighted mode.

9. The method for continuous sign language recognition based on dual-stream multi-fusion and frame enhancement according to claim 1, characterized in that: Obtaining a target recognition result according to the first target weighted mode and the second target weighted mode includes the following steps: Summing the first target weighted mode and the second target weighted mode to obtain a second intermediate weighted mode; The second intermediate weighted mode is input into the linear layer to obtain the target recognition result.

10. A dual-stream multi-fusion continuous sign language recognition device based on frame enhancement, characterized in that: include: The first module is used to obtain the bone heat map based on the RGB image; The second module is used to extract features from the RGB image and the bone heat map through a frame enhancement module and a bidirectional long short-term memory network to obtain a first extracted feature of the RGB image stream and a second extracted feature of the heat map stream; wherein the frame enhancement module includes a spatial guidance submodule and a motion guidance submodule; after passing through the spatial guidance submodule, spatial guidance features are obtained through self-attention calculation; after passing through the motion guidance submodule, motion guidance features are obtained through a convolution layer and forward difference operation; after passing through the frame enhancement module, the RGB image stream and the heat map stream are interacted through a bidirectional connection; A third module is configured to obtain a first target weighted mode and a second target weighted mode according to the first extracted features and the second extracted features; The fourth module is used to obtain a target recognition result according to the first target weighted mode and the second target weighted mode.

Images