Isolated word sign language recognition method and device, equipment and medium

Abstract

The invention relates to the technical field of sign language recognition, in particular to an isolated word sign language recognition method and device, equipment and a medium, and the method comprises the steps: collecting an RGB video of an isolated word sign language action of a sign language user, the RGB video comprising a video collected from the front of the sign language user and a video collected from the side of the sign language user; performing down-sampling based on the RGB video to obtain a front sign language video frame sequence and a side sign language video frame sequence; extracting a first spatial feature based on the front sign language video frame sequence, and extracting a second spatial feature based on the side sign language video frame sequence; based on the first spatial feature and the second spatial feature, obtaining a fused spatial feature; performing time sequence modeling on the fused spatial features to obtain target classification features; based on the target classification features and the sign language recognition model, the isolated word sign language recognition result is obtained, feature extraction is performed through the double-view sign language video, the visual occlusion problem of single-view sign language recognition can be avoided, and then the accuracy of isolated word sign language recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language recognition, and in particular to a sign language isolated word recognition method, device, equipment and medium. BACKGROUND

[0002] Sign language recognition includes static sign language recognition and dynamic sign language recognition, wherein the dynamic sign language recognition further includes isolated word recognition and continuous word sentence recognition. An isolated word refers to a discrete and independent sign language word in time and space, which is usually composed of three parts: a preparation action, an effective action and an ending action. For example, the isolated word sign language word "moving house" is composed of a preparation action of lifting both hands, an effective action of moving both hands in a zigzag shape from left to right, and an ending action of lowering the hands. The isolated word sign language words have no context association, and need to be recognized separately. The isolated word sign language recognition is commonly used in the fields of sign language translation system and communication tools for hearing-impaired people.

[0003] However, the existing isolated word sign language recognition cannot effectively recognize the specific meaning in the case of visual occlusion and the like, and has the problem of low recognition accuracy. SUMMARY

[0004] In view of the above problems, the present application provides an isolated word sign language recognition method, device, equipment and medium which overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, the present application provides an isolated word sign language recognition method, comprising: collecting an RGB video of an isolated word sign language action of a sign language user, wherein the RGB video includes a video collected from the front of the sign language user and a video collected from the side of the sign language user; performing down-sampling based on the RGB video to obtain a front sign language video frame sequence and a side sign language video frame sequence; extracting a first spatial feature based on the front sign language video frame sequence and a second spatial feature based on the side sign language video frame sequence; obtaining a fused spatial feature based on the first spatial feature and the second spatial feature; performing time sequence modeling on the fused spatial feature to obtain a target classification feature; obtaining an isolated word sign language recognition result based on the target classification feature and a sign language recognition model.

[0006] Preferably, the down-sampling based on the RGB video to obtain the front sign language video frame sequence and the side sign language video frame sequence comprises: converting each frame of image of the RGB video into a grayscale image; determining a starting frame of the isolated word sign language action based on the grayscale image; Sampling with the starting frame as the starting point of downsampling, to obtain a front-facing time-aligned biview sign language video frame sequence and a side-facing time-aligned biview sign language video frame sequence.

[0007] Preferably, determining the starting frame of the isolated word sign language action based on the grayscale image comprises: Based on the grayscale image, Gaussian blur is used to suppress high-frequency noise to obtain a pixel matrix of the grayscale image. Based on the pixel matrix, the motion intensity map corresponding to each frame of image is calculated. Based on the motion intensity map, the motion intensity of each frame of image is determined. Based on the motion intensity of each frame of image, the starting frame of the isolated word sign language action is determined.

[0008] Preferably, determining the starting frame of the isolated word sign language action based on the motion intensity of each frame of image comprises: Judging whether the motion intensity of the continuous frame of image is greater than the respective adaptive intensity threshold value; If yes, the first frame of image of the continuous frame of image is taken as the starting frame of the isolated word sign language action.

[0009] Preferably, the adaptive intensity threshold value is determined in the following manner: ; For the median function, For the motion intensity sequence determined based on the motion intensity of each frame of image, For the motion intensity of the corresponding frame of image, For the adaptive intensity threshold value corresponding to the corresponding frame of image.

[0010] Preferably, based on the first spatial feature and the second spatial feature, the fusion spatial feature is obtained, comprising: The first spatial feature and the second spatial feature are fused by using a multi-modal feature fusion method to obtain a fusion spatial feature, the multi-modal feature fusion method comprising a cross-attention mechanism deep neural network fusion, a deformable attention mechanism deep neural network fusion and a gated neural network fusion, and the fusion spatial feature is obtained according to the following calculation formula: ; Wherein, The cross-attention mechanism deep neural network fusion, The deformable attention mechanism deep neural network fusion, The gated neural network fusion, The first spatial feature, The second spatial feature, The fusion spatial feature.

[0011] Preferably, the fusion spatial features are time-series modeled to obtain target classification features, including: The fusion spatial features are input into a deep time-series neural network to output target classification features, and the deep time-series neural network is specifically a Transformer encoder.

[0012] In a second aspect, the present application further provides an isolated word sign language recognition device, including: A collection module is configured to collect an RGB video of isolated word sign language actions of a sign language user, wherein the RGB video includes a video collected from the front of the sign language user and a video collected from the side of the sign language user. A sampling module is configured to perform down-sampling based on the RGB video to obtain a front sign language video frame sequence and a side sign language video frame sequence. An extraction module is configured to extract first spatial features based on the front sign language video frame sequence and extract second spatial features based on the side sign language video frame sequence. A fusion module is configured to obtain fusion spatial features based on the first spatial features and the second spatial features. A time-series modeling module is configured to time-series model the fusion spatial features to obtain target classification features. A obtaining module is configured to obtain isolated word sign language recognition results based on the target classification features and a sign language recognition model.

[0013] In a third aspect, the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method steps of the first aspect when executing the program.

[0014] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program executable by a processor to implement the method of the first aspect.

[0015] One or more technical solutions in the embodiments of the present application have at least the following technical effects or advantages: The application provides a sign language isolated word recognition method, comprising the following steps: collecting an RGB video of sign language isolated word actions of a sign language user, wherein the RGB video comprises a video collected from the front of the sign language user and a video collected from the side of the sign language user; performing downsampling based on the RGB video to obtain a front sign language video frame sequence and a side sign language video frame sequence; extracting first spatial features based on the front sign language video frame sequence and extracting second spatial features based on the side sign language video frame sequence; obtaining fused spatial features based on the first spatial features and the second spatial features; performing time sequence modeling on the fused spatial features to obtain target classification features; and obtaining a sign language isolated word recognition result based on the target classification features and a sign language recognition model. The feature extraction is performed through double-view sign language videos, the visual occlusion problem of single-view sign language recognition can be avoided, and the accuracy of sign language isolated word recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings. Figure 1 A step flow diagram of the sign language isolated word recognition method in the embodiment of the application is shown; Figure 2 A whole process diagram of the sign language isolated word recognition in the embodiment of the application is shown; Figure 3 A structure diagram of the sign language isolated word recognition device in the embodiment of the application is shown; Figure 4 A schematic diagram of a computer device for implementing the sign language isolated word recognition method in the embodiment of the application is shown. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be accurately conveyed to those skilled in the art.

[0018] Embodiment one: The embodiment of the present application provides a sign language isolated word recognition method, as shown in the accompanying drawings, comprising the following steps: Figure 1 S101, collecting an RGB video of sign language isolated word actions of a sign language user, wherein the RGB video comprises a video collected from the front of the sign language user and a video collected from the side of the sign language user; ​S102, based on the RGB video, downsampling is performed to obtain a front sign language video frame sequence and a side sign language video frame sequence; S103, based on the front sign language video frame sequence, a first spatial feature is extracted, and based on the side sign language video frame sequence, a second spatial feature is extracted; S104, based on the first spatial feature and the second spatial feature, a fusion spatial feature is obtained; S105, the fusion spatial feature is subjected to time sequence modeling to obtain a target classification feature; S106, based on the target classification feature and a sign language recognition model, an isolated word sign language recognition result is obtained.

[0019] In a specific embodiment, by deploying video acquisition devices on the front and side of the sign language user, video acquisition is performed through the dual perspectives of the front and side, which can avoid the problem of being unable to effectively recognize due to occlusion in one perspective. Therefore, S101, an RGB video of the isolated word sign language action of the sign language user is acquired, and the RGB video includes a video acquired from the front of the sign language user and a video acquired from the side of the sign language user.

[0020] Specifically, the acquired RGB video is wherein, corresponding to the first sign language video sample, corresponding to the front sign language video of the first sign language video sample, corresponding to the side sign language video of the first sign language video sample. The vocabulary label corresponding thereto is wherein, , is a set of all isolated word labels.

[0021] Each RGB sign language video can be represented as a 4-dimensional tensor wherein, is the number of frames of the RGB sign language video, is the number of channels, Next, S102 is performed, and based on the RGB video, downsampling is performed to obtain a front sign language video frame sequence and a side sign language video frame sequence.

[0022] Specifically, each frame of the RGB video is converted into a grayscale image; Based on the grayscale image, the starting frame of the isolated word sign language action is determined; The starting frame is taken as the starting point of downsampling to obtain a front sign language video frame sequence and a side sign language video frame sequence that are time sequence aligned in dual perspectives.

[0023] In a specific embodiment, a starting frame is determined first, and then down-sampling is performed based on the starting frame.

[0024] In determining the starting frame, the following method is used: Based on the gray image, Gaussian blur is used to suppress Gaussian noise, and a pixel matrix of the gray image is obtained. Based on the pixel matrix, a motion intensity map corresponding to each frame of image is calculated. Based on the motion intensity map, the motion intensity of each frame of image is determined. Based on the motion intensity of each frame of image, the starting frame of the isolated word sign language action is determined.

[0025] In using Gaussian blur to suppress Gaussian noise, the obtained pixel matrix of the gray image is wherein is the serial number of the frame.

[0026] Then, based on the pixel matrix, a motion intensity map corresponding to each frame of image is calculated, that is, Specifically, the difference between the pixel matrices of the front and rear gray images is obtained, and the motion intensity map is calculated according to the following formula:

[0027] Then, based on the motion intensity map, the motion intensity of each frame of image is determined. Specifically, the arithmetic mean of all pixel points is calculated as the motion intensity of each frame of image Thus, the motion intensity sequence of the entire sign language video is obtained, that is, The calculation formula of the motion intensity is as follows:

[0028] After obtaining the motion intensity of each frame of image, based on the motion intensity of each frame of image, the starting frame of the isolated word sign language action is determined, and the specific determination method is as follows: It is judged whether the motion intensity of the continuous frame image is greater than the respective adaptive intensity threshold value; If yes, the first frame image of the continuous frame image is taken as the starting frame of the isolated word sign language action.

[0029] That is, the continuous frame image is searched from the 1st frame backward, and if the continuous frame starting from one frame If the motion intensity of the frame is greater than the adaptive intensity threshold value of the respective frame , then the frame is the starting frame. The specific calculation formula is as follows:

[0030] an adaptive intensity threshold value for the corresponding frame.

[0031] The adaptive intensity threshold value is calculated in the following manner:

[0032] wherein, is a median function, is a motion intensity sequence determined based on the motion intensity of each frame image, is the motion intensity of the corresponding frame image, is the adaptive intensity threshold value corresponding to the corresponding frame image.

[0033] After obtaining the starting frame of the isolated word sign language action in the above manner, sampling is performed with the starting frame as the starting point of downsampling, and if the sampling interval is 5 frames, 16 frames are continuously sampled to obtain a front sign language video frame sequence and a side sign language video frame sequence , and 16 is the number of frames.

[0034] During the sampling process, if the number of sampled frames is insufficient, the last frame image is used for padding.

[0035] Due to the characteristics of isolated words, the entire sign language process image frame sampling of the isolated word is effectively obtained by accurately confirming the starting frame, which facilitates the accuracy of subsequent recognition.

[0036] Next, S103 is performed, and a first spatial feature is extracted based on the front sign language video frame sequence, and a second spatial feature is extracted based on the side sign language video frame sequence.

[0037] In a specific implementation, the feature extraction is specifically performed by a feature extractor. For example, a front feature extractor extracts the front sign language feature, i.e., the first spatial feature, and the specific formula is as follows: A side feature extractor extracts the side sign language feature, i.e., the second spatial feature, and the specific formula is as follows: , is the dimension of the feature vector.

[0038] The specific formula is as follows:

[0039] wherein, the feature extractor adopts Swin ViT.

[0040] Then, S104 is performed, and a fused spatial feature is obtained based on the first spatial feature and the second spatial feature.

[0041] Specifically, the first spatial feature and the second spatial feature are fused by using a multi-modal feature fusion method to obtain a fused spatial feature, the multi-modal feature fusion method includes cross-attention mechanism deep neural network fusion, deformable attention mechanism deep neural network fusion, and gate neural network fusion, and the fused spatial feature is obtained according to the following calculation formula: ; wherein, is the cross-attention mechanism deep neural network fusion, is the deformable attention mechanism deep neural network fusion, is the gate neural network fusion, is the first spatial feature, is the second spatial feature, is the fused spatial feature.

[0042] The fusion means is specifically a front-view guided early fusion strategy FvGFF (Front-view Guided Early Fusion) mechanism. The mechanism takes advantage of the characteristics that the front view plays a major role and the side view plays a supplementary role in the dual-view sign language video, and can effectively fuse the dual-view. The cross-attention mechanism deep neural network can preliminarily fuse the first spatial feature and the second spatial feature, the deformable attention mechanism deep neural network ensures the semantic consistency between the dual views, and the gate neural network emphasizes the guiding role of the front view features and the supplementary role of the side view features through the gate mechanism.

[0043] After obtaining the fused spatial feature, S105 is performed to model the fused spatial feature in time sequence to obtain a target classification feature.

[0044] Specifically, the fused spatial feature is input into a deep time sequence neural network, and the target classification feature is output, and the deep time sequence neural network is specifically a Transformer encoder.

[0045] The deep time sequence neural network is used to model the fused spatial feature in the time dimension to better understand the input fused spatial feature and obtain the target classification feature, and the deep time sequence neural network is specifically as follows:

[0046] Finally, S106 is performed to obtain a word isolation sign language recognition result based on the target classification feature and a sign language recognition model.

[0047] In a specific embodiment, the sign language recognition model is obtained based on training, specifically by collecting a large number of word isolation sign language videos and training in combination with corresponding word isolation labels. The sign language recognition model finally obtains a word isolation sign language recognition result.

[0048] Since sign language content is obtained by combining multiple isolated sign words, the meaning of the entire sign language sentence can be obtained by combining them according to grammar based on the results of isolated sign word recognition.

[0049] like Figure 2 The diagram shows the overall process of isolated word sign language recognition. First, a frontal sign language video is acquired. And side sign language video Next, downsampling is performed by determining the starting frame, thereby obtaining the frontal sign language video frame sequence. and side sign language video frame sequence Then, a frontal feature extractor is used to extract frontal sign language features, i.e., first-space features. A side feature extractor is used to extract side sign language features, i.e., second space features. Next, the first spatial features Fusion of second spatial features The fusion process includes deep neural network fusion using cross-attention mechanisms, deep neural network fusion using deformable attention mechanisms, and fusion using gated neural networks. After these fusion methods, fusion spatial features are obtained. Then, through temporal modeling, specifically using deep temporal neural networks, target classification features are obtained. Finally, the target classification feature is input into the fully connected layer of the trained sign language recognition model, and the final output is the isolated word sign language recognition result.

[0050] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages: This invention provides a method for isolated word sign language recognition, comprising: acquiring RGB video of isolated word sign language actions of a sign language user, the RGB video including video captured from the front of the sign language user and video captured from the side of the sign language user; downsampling based on the RGB video to obtain a frontal sign language video frame sequence and a side sign language video frame sequence; extracting a first spatial feature based on the frontal sign language video frame sequence and extracting a second spatial feature based on the side sign language video frame sequence; obtaining a fused spatial feature based on the first spatial feature and the second spatial feature; performing temporal modeling on the fused spatial feature to obtain a target classification feature; and obtaining an isolated word sign language recognition result based on the target classification feature and a sign language recognition model. By using dual-view sign language video for feature extraction, the visual occlusion problem of single-view sign language recognition can be avoided, thereby improving the accuracy of isolated word sign language recognition.

[0051] Example 2: Based on the same inventive concept, embodiments of the present invention also provide an isolated word sign language recognition device, such as... Figure 3 As shown, it includes: The collection module 301 is configured to collect an RGB video of a sign language user's isolated word sign language action, wherein the RGB video includes a video collected from the front of the sign language user and a video collected from the side of the sign language user. The sampling module 302 is configured to perform downsampling based on the RGB video to obtain a front sign language video frame sequence and a side sign language video frame sequence. The extraction module 303 is configured to extract first spatial features based on the front sign language video frame sequence and extract second spatial features based on the side sign language video frame sequence. The fusion module 304 is configured to obtain fused spatial features based on the first spatial features and the second spatial features. The time sequence modeling module 305 is configured to perform time sequence modeling on the fused spatial features to obtain target classification features. The obtaining module 306 is configured to obtain an isolated word sign language recognition result based on the target classification features and a sign language recognition model.

[0052] In an optional implementation, the sampling module 302 is configured to: convert each image of the RGB video into a grayscale image; determine a starting frame of the isolated word sign language action based on the grayscale image; perform sampling with the starting frame as a starting point of downsampling to obtain a front sign language video frame sequence and a side sign language video frame sequence that are time-aligned in dual-view.

[0053] In an optional implementation, the sampling module 302 is further configured to: suppress high-frequency noise by using Gaussian blur based on the grayscale image to obtain a pixel matrix of the grayscale image; calculate a motion intensity map corresponding to each image based on the pixel matrix; determine a motion intensity of each image based on the motion intensity map; determine the starting frame of the isolated word sign language action based on the motion intensity of each image.

[0054] In an optional implementation, the sampling module 302 is further configured to: determine whether motion intensities of consecutive images are all greater than respective adaptive intensity thresholds; if yes, take a first image of the consecutive images as the starting frame of the isolated word sign language action.

[0055] In an optional implementation, the adaptive intensity threshold is determined in the following manner: ; a median function is taken, a motion intensity sequence determined based on the motion intensity of each frame image, a motion intensity of a corresponding frame image, an adaptive intensity threshold corresponding to the corresponding frame image.

[0056] In an optional implementation, the fusion module 304 is configured to: fuse the first spatial feature and the second spatial feature by using a multi-modal feature fusion method to obtain a fused spatial feature, the multi-modal feature fusion method including cross-attention mechanism deep neural network fusion, deformable attention mechanism deep neural network fusion, and gate neural network fusion, and the fused spatial feature is obtained according to the following calculation formula: ; wherein, the cross-attention mechanism deep neural network fusion, the deformable attention mechanism deep neural network fusion, the gate neural network fusion, the first spatial feature, the second spatial feature, the fused spatial feature.

[0057] In an optional implementation, the obtaining module 306 is configured to: input the fused spatial feature into a deep time sequence neural network to output a target classification feature, and the deep time sequence neural network is specifically a Transformer encoder.

[0058] Embodiment three: Based on the same inventive concept, the embodiments of the present application provide a computer device, as shown in Figure 4 the memory 404, the processor 402, and a computer program stored in the memory 404 and executable on the processor 402, and the processor 402 implements the steps of the above-mentioned isolated word sign language recognition method when executing the program.

[0059] wherein, in Figure 4In this particularized embodiment, a bus architecture (represented by bus 400) can include any number of interconnecting buses and bridges needed to link various circuitry, including one or more processors represented by processor 402 and memory represented by memory 404. Bus 400 can also link various other circuitry, such as peripheral devices, voltage regulators, and power management circuitry, all of which are well known in the art, and therefore, not described further. Bus interface 406 provides an interface between bus 400 and receiver 401 and transmitter 403. Receiver 401 and transmitter 403 can be the same component, i.e., a transceiver, providing a means for communicating with various other apparatus over the transmission medium. Processor 402 is responsible for managing the bus 400 and general processing, while the memory 404 can be used for storing data used by the processor 402 in executing operational processes.

[0060] Embodiment Four Based on the same inventive concept, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the above-described isolated word sign language recognition method.

[0061] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present application is not intended to be limited to any particular programming language. It will be appreciated that there are many programming languages that can be used to implement the teachings herein, and any such programming language can be used in connection with the various aspects of the present application.

[0062] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0063] Similarly, it is to be understood that the embodiments of the present application can be used in the exact form disclosed herein, or with minor modifications, and the present application is not limited to the exact form disclosed herein. Accordingly, it is not intended that the application be limited, in all aspects, to the specific embodiments discussed hereinabove. Furthermore, the teachings of the present application can be used to construct other specific embodiments and the present application is not limited to the specific forms disclosed herein. Rather, these embodiments are provided as examples of implementing the application. Accordingly, the application is not to be limited in scope by the specific embodiments described herein, but rather is to be construed broadly within the spirit and the scope of the claims submitted herewith.

[0064] Those skilled in the art will appreciate that the modules in the apparatus in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and furthermore can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and all the processes or units of any method or apparatus disclosed thus can be adopted. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar function.

[0065] Furthermore, those skilled in the art will appreciate that a combination of features of different embodiments means within the scope of the present application and forms different embodiments, although some embodiments herein include certain features rather than others included in other embodiments. For example, in the DETAILED DESCRIPTION, any one of the claimed embodiments can be used in any combination.

[0066] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combination thereof. Those skilled in the art will appreciate that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the isolated word sign language recognition apparatus and the computer device according to the embodiments of the present application. The present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0067] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices or means can be listed, comprising means which can be implemented by one and the same hardware item. The use of the word "a" or "an" does not exclude the presence of a plurality of such elements, nor does it imply that a single element is to be used.

Claims

1. A method for identifying isolated words in sign language, characterized in that, include: RGB video of isolated word sign language actions of sign language users is acquired, the RGB video including video captured from the front of the sign language user and video captured from the side of the sign language user; Based on the RGB video, downsampling is performed to obtain a frontal sign language video frame sequence and a side sign language video frame sequence; Based on the frontal sign language video frame sequence, a first spatial feature is extracted, and based on the side sign language video frame sequence, a second spatial feature is extracted. Based on the first spatial feature and the second spatial feature, a fused spatial feature is obtained; Temporal modeling is performed on the fused spatial features to obtain target classification features; Based on the target classification features and sign language recognition model, the isolated word sign language recognition results are obtained.

2. The method as described in claim 1, characterized in that, Based on the RGB video, downsampling is performed to obtain a frontal sign language video frame sequence and a side sign language video frame sequence, including: Each frame of the RGB video is converted into a grayscale image; Based on the grayscale image, determine the starting frame of the isolated word sign language action; Using the starting frame as the starting point for downsampling, sampling is performed to obtain a frontal sign language video frame sequence and a side sign language video frame sequence with dual-view temporal alignment.

3. The method as described in claim 2, characterized in that, Based on the grayscale image, the starting frame of the isolated word sign language action is determined, including: Based on the grayscale image, Gaussian blur is used to suppress high-frequency noise, thereby obtaining the pixel matrix of the grayscale image; Based on the pixel matrix, calculate the motion intensity map corresponding to each frame of the image; Based on the motion intensity map, the motion intensity of each frame of the image is determined; The starting frame for isolated word sign language actions is determined based on the motion intensity of each frame.

4. The method as described in claim 3, characterized in that, Based on the motion intensity of each frame, the starting frame for the isolated word sign language action is determined, including: Determine whether the motion intensity of consecutive frames is greater than their respective adaptive intensity thresholds; If so, the first frame of the continuous frame images is taken as the starting frame of the isolated word sign language action.

5. The method as described in claim 4, characterized in that, The adaptive intensity threshold is determined as follows: ； To obtain the median function, This is a sequence of motion intensity determined based on the motion intensity of each frame. The motion intensity of the corresponding frame image. This is the adaptive intensity threshold corresponding to the corresponding frame image.

6. The method as described in claim 1, characterized in that, Based on the first spatial feature and the second spatial feature, a fused spatial feature is obtained, including: A multimodal feature fusion method is used to fuse the first spatial features and the second spatial features to obtain fused spatial features. The multimodal feature fusion method includes cross-attention mechanism deep neural network fusion, deformable attention mechanism deep neural network fusion, and gated neural network fusion, specifically calculated according to the following formula: ； in, For deep neural network fusion with cross-attention mechanism, Fusion of deep neural networks with deformable attention mechanisms For gated neural network fusion, As the first spatial feature, As a second spatial feature, To integrate spatial features.

7. The method as described in claim 1, characterized in that, Temporal modeling is performed on the fused spatial features to obtain target classification features, including: The fused spatial features are input into a deep temporal neural network, and the target classification features are output. The deep temporal neural network is specifically a Transformer encoder.

8. A sign language recognition device for isolated words, characterized in that, include: The acquisition module is used to acquire RGB video of isolated word sign language actions of sign language users. The RGB video includes video acquired from the front of the sign language user and video acquired from the side of the sign language user. The sampling module is used to downsample the RGB video to obtain a frontal sign language video frame sequence and a side sign language video frame sequence. The extraction module is used to extract a first spatial feature based on the frontal sign language video frame sequence and to extract a second spatial feature based on the side sign language video frame sequence. The fusion module is used to obtain fused spatial features based on the first spatial features and the second spatial features; The temporal modeling module is used to perform temporal modeling on the fused spatial features to obtain target classification features; The module is used to obtain isolated word sign language recognition results based on the target classification features and the sign language recognition model.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images