Event stream sign language translation method based on state space
By introducing CNN module, Mamba state space module and Temporal Conv module into sign language translation technology, combining sign language embedding and prompt modules, the problem that a single network architecture in the existing technology cannot effectively capture sign language features is solved, and more comprehensive feature extraction and higher translation performance is achieved.
Patent Information
- Application Number
- CN202411665809.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-11
AI Technical Summary
The existing sign language translation technology mainly relies on a single network architecture and cannot effectively exert the advantages of different architectures, resulting in limitations in the feature extraction process and the inability to fully capture the local details and long-distance dependencies of sign language.
The event flow sign language translation method based on state space is adopted, and the local-global visual feature representation is extracted by introducing the CNN module and the Mamba state space module, and the Temporal Conv module is used to fusion of timing features, combining the sign language embedding module and prompt module to achieve the alignment of visual features and language features and the improvement of model performance.
Effectively extract local details and long-distance dependencies of sign language, comprehensively and fully extract data information, improve model performance, reduce redundant features, and improve translation effect.
Smart Images
Figure CN119600683A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of sign language translation, and in particular to a state space-based event stream sign language translation method. Background Art
[0002] Recently, bio-inspired event cameras have attracted increasing attention due to their unique characteristics in terms of high dynamic range, low energy cost, sparse spatial but dense temporal resolution, and low latency. For the imaging mechanism, each pixel within the event camera operates independently and asynchronously, which is very different from traditional visible light cameras (RGB cameras) that produce synchronous frame outputs. The binary event / pulse signal is recorded only when the brightness change of each pixel exceeds a certain threshold. Some computer vision tasks involve event cameras for event-based or event-enhanced learning, such as object detection, tracking, action recognition, and sign language translation.
[0003] With the rapid development of deep learning, artificial intelligence has received widespread attention, among which sign language translation has received increasing attention. It helps the deaf communicate with non-sign language users by converting sign language videos into natural language text, greatly improving the social participation and quality of life of the deaf community.
[0004] At present, some research works mainly focus on sign language translation of RGB cameras, including convolutional neural networks (CNN), spiking neural networks (SNN) and Transformer networks. Although these methods have achieved good translation results through different feature representations and learning processes. However, due to the use of RGB cameras, sign language translation still has limitations in the following aspects: Currently, sign language translation is limited to using a single network architecture for feature learning. These methods have their advantages in the process of feature extraction, but they also have their own limitations. For example, CNN focuses on the details of local features and cannot capture long-distance dependencies well. Transformer better models long-distance relationships, but its self-attention has a high computational complexity and requires more computing resources. A single network architecture cannot effectively play the advantages of different architectures.
[0005] Based on this, the present invention designs an event flow sign language translation method based on state space to solve the above problems. Summary of the invention
[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides an event flow sign language translation method based on state space.
[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0008] A state-space-based event stream sign language translation method comprises the following steps:
[0009] Step S1, input the event stream and convert it into an event image, and input the event image into the Mamba state space module to extract local-global visual feature representation;
[0010] Step S2: The local spatial feature representation F of the CNN module in the last layer of the Mamba state space module C The global state space feature representation F of the last layer M Input into the Temporal Conv module for temporal feature fusion;
[0011] Step S3: Then, the fused temporal features are sent to the sign language embedding module to align the visual features to the language features;
[0012] Step S4: Then, use the prompt module to construct a prompt sentence, input the prompt sentence and the aligned features into the LLM language module to generate a sign language sentence.
[0013] Furthermore, step S1 specifically includes the following steps:
[0014] S11, converting the event stream into an event image;
[0015] S12, extracting initial visual features of event images through the StemNet network;
[0016] S13, then input the initial visual features into the CNN module to extract local spatial feature representation;
[0017] S14. Then, the Mamba state space module is used to extract information from the local spatial feature representation from the CNN module from both forward and backward directions to obtain the global state space feature representation.
[0018] Furthermore, the method used to divide the event stream data is: the event source data stream is a four-tuple {X, Y, T, P}, where X and Y represent the spatial position coordinates, T represents time, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the spatial position without an event is marked as 0. It is divided according to time t to form segments of size X×Y×t, with a total of T / t segments.
[0019] Furthermore, in step S14, the local-global visual feature representation method is: the local spatial feature representation F from the CNN module C Input into the Mamba state space module, use the C2M module to convert the feature map into a feature sequence, pass through a normalization layer, and then use two linear layers to project it into features x and z; use a 1D convolutional layer and SiLU activation function to obtain x forand x back ; The forward SSM and backward SSM branches are used for bidirectional context perception to generate the forward state space SSM forward and the backward state space SSM backward ; Then the feature z is activated by SiLU function and SSM forward Features and SSM backward The features are added together after inner product operation and input into the projection layer to obtain the global state space feature representation F M ; Finally, the global state space feature representation F M The feature sequence is converted into a feature map through the M2C (global to local) module and then combined with the local spatial feature representation F C Spliced together, they are input into the next layer of CNN module and Mamba state space module.
[0020] Furthermore, step S14 adopts formula (1), formula (2), and formula (3):
[0021] x=Linear(Norm(F C )), z=Linear(Norm(F C )) Formula (1)
[0022] x for =SiLU(Conv(x)), x back =SiLU(Conv(x)) Formula (2)
[0023] F M =Linear(SSM forward (x for )⊙SiLU(z)+SSM backward (x back )⊙SiLU(z))Formula (3).
[0024] Furthermore, step S4 specifically includes the following steps:
[0025] S41, the prompt module constructs a prompt sentence for each event stream video, and inputs the prompt sentence and the aligned features into the LLM language module;
[0026] S42, LLM language module is the LLM language decoder, which uses the cross entropy loss function to calculate the distance between the true sign language sentence and the generated sign language sentence, and generates the final sign language sentence for each event stream video.
[0027] Furthermore, step S42 adopts formula (4) and formula (5):
[0028] S={p( <bos>), p(w1), p(w2),..., p(w U ),p( <eos>)} Formula (4)
[0029]
[0030] The first special word <bos>Indicates the beginning of a sentence, the last special word <eos>Indicates the end of the sentence, p(w) represents the probability distribution of the generated words, U represents the number of words in the sign language sentence, S represents the sign sentence Sentence, L represents the loss function Loss, u represents the subscript of the number of sign language words, s u Represents the sentence consisting of the current u words, w1, w2, w u They represent the first, second, and uth word in the sentence respectively, and p represents the conditional probability.
[0031] Compared with the prior art, the present invention has the following beneficial effects: 1. By introducing the CNN module and the Mamba state space module, the present invention can effectively extract local detail texture information of the data while capturing the long-distance dependencies of the data, so that the model can fully and comprehensively extract data information and improve the performance of the model.
[0032] 2. The present invention adopts a Temporal Conv module based on convolution and pooling, which can fuse the temporal information of visual feature Tokens, reduce the number of visual feature Tokens, and avoid the problem of obtaining suboptimal solutions due to the inclusion of redundant features.
[0033] 3. The present invention adopts a sign language embedding module based on linear projection, which can align the fused visual feature Token with the language feature, thereby realizing the bridge between the visual feature and the language feature.
[0034] 4. The present invention introduces a prompt module (prompt statement), which can generate corresponding prompt statements according to the data content, guide the model to pay attention to data details, and further improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 is a flow chart of the method of the present invention;
[0037] Figure 2 It is a schematic diagram of the model framework of the present invention. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] Glossary:
[0040] Sign language truth statements: obtained from the sign language dataset. The sign language event stream and the sign language truth statements corresponding to each event stream are stored in the sign language dataset.
[0041] In some embodiments, please refer to the drawings of the specification. Figure 1-Figure 2 , an event stream sign language translation method based on state space, comprising the following steps:
[0042] Step S1: Input the event stream and convert it into an event image. The event image is input into the Mamba state space module to extract local-global visual feature representation.
[0043] The specific steps include:
[0044] S11, converting the event stream into an event image;
[0045] S12, extracting initial visual features of event images through the StemNet network;
[0046] S13, then input the initial visual features into the CNN module to extract local spatial feature representation;
[0047] S14. Then, the Mamba state space module is used to extract information from the local spatial feature representation from the CNN module from both forward and backward directions to obtain the global state space feature representation.
[0048] The method used to divide the event stream data is: the event source data stream is a four-tuple {X, Y, T, P}, where X and Y represent the spatial position coordinates, T represents time, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the spatial position without an event is marked as 0. It is divided according to time t to form segments of size X×Y×t, with a total of T / t segments.
[0049] In step S14, the local-global visual feature representation method is: the local spatial feature representation F from the CNN module C Input into the Mamba state space module, use the C2M module (from CNN module to Mamba state space module, representing from local to global) to convert the feature map into a feature sequence, pass through a normalization layer, and then use two linear layers to project it into features x and z, both of which are features obtained through the linear layer; use a 1D convolutional layer and SiLU activation function to obtain x for and x back The forward SSM and backward SSM branches are used for bidirectional context perception to generate the forward state space SSM forward (forward process in state space) and backward state space SSM backward (Backward process in state space); then the feature z is activated by SiLU function and SSM forward Features and SSM backward The features are added together after inner product operation and input into the projection layer to obtain the global state space feature representation F M .
[0050] As shown in formula (1), formula (2) and formula (3):
[0051] x=Linear(Norm(F C )), z=Linear(Norm(F C )) Formula (1)
[0052] x for =SiLU(Conv(x)), x back =SiLU(Conv(x)) Formula (2)
[0053] F M =Linear(SSM forward (x for )⊙SiLU(z)+SSM backward (x back )⊙SiLU(z))Formula (3)
[0054] Among them, SSM forward and SSM backward Both are SSM functions, but they process data in different directions;
[0055] Finally, the global state space feature representation F M The feature sequence is converted into a feature map through the M2C module (from the Mamba state space module to the CNN module, representing from global to local) and then combined with the local space feature representation F C Spliced together, they are input into the next layer of CNN module and Mamba state space module.
[0056] Step S2: The local spatial feature representation F of the CNN module in the last layer of the Mamba state space module C The global state space feature representation F of the last layer M Input into the Temporal Conv module for temporal feature fusion. The temporal information of the features is fused through the temporal convolution module, while reducing the number of visual feature tokens.
[0057] Step S3: Then, the fused temporal features are sent to the Sign Embedding module to align the visual features with the language (text) features.
[0058] Step S4: Then, use the prompt module (prompt prompt sentence) to construct a prompt sentence, input the prompt sentence and the aligned features into the LLM language module together to generate a sign language sentence.
[0059] The specific steps include:
[0060] S41, the prompt module constructs a prompt sentence for each event stream video, and inputs the prompt sentence and the aligned features into the LLM language module;
[0061] S42, LLM language module is the LLM language decoder, which uses the cross entropy loss function to calculate the distance between the sign language truth statement and the generated sign language statement, and generates the final sign language statement for each event stream video. As shown in formula (4) and formula (5):
[0062] S={p( <bos>), p(w1), p(w2),..., p(w U ),p( <eos>)} Formula (4)
[0063]
[0064] The first special word <bos>Indicates the beginning of a sentence, the last special word <eos>Indicates the end of the sentence, p(w) represents the probability distribution of the generated words, U represents the number of words in the sign language sentence, S represents the sign sentence Sentence, L represents the loss function Loss, u represents the subscript of the number of sign language words, s u Represents the sentence consisting of the current u words, w1, w2, w u Respectively represent the first, second, and uth words in the sentence, and p represents the first word in word w u under the conditions of s u probability.
[0065] By introducing the CNN module and the Mamba state space module, the present invention can effectively extract local detail texture information of the data while capturing the long-distance dependencies of the data, so that the model can fully and comprehensively extract data information and improve the performance of the model.
[0066] The present invention adopts a Temporal Conv module based on convolution and pooling, which can fuse the temporal information of visual feature Tokens, reduce the number of visual feature Tokens, and avoid the problem of obtaining a suboptimal solution due to the inclusion of redundant features.
[0067] The present invention adopts a sign language embedding module based on linear projection, which can align the fused visual feature Token with the language feature, thereby realizing the bridge between the visual feature and the language feature.
[0068] The present invention introduces a prompt module (prompt statement), which can generate corresponding prompt statements according to the data content, guide the model to pay attention to data details, and further improve the performance of the model.
[0069] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos> < / bos> < / eos> < / bos> < / eos> < / bos> < / eos> < / bos>
Claims
1. A state-space-based event-flow sign language translation method, characterized in that: The following steps are involved: Step S1, input the event stream and convert it into an event image, and input the event image into the Mamba state space module to extract local-global visual feature representation; Step S2: The local spatial feature representation F of the CNN module in the last layer of the Mamba state space module c The global state space feature representation F of the last layer M Input into the Temporal Conv module for temporal feature fusion; Step S3: Then, the fused temporal features are sent to the sign language embedding module to align the visual features to the language features; Step S4: Then, use the prompt module to construct a prompt sentence, input the prompt sentence and the aligned features into the LLM language module to generate a sign language sentence.
2. The event flow sign language translation method based on state space according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S11, converting the event stream into an event image; S12, extracting initial visual features of event images through the StemNet network; S13, then input the initial visual features into the CNN module to extract local spatial feature representation; S14. Then, the Mamba state space module is used to extract information from the local spatial feature representation from the CNN module from both forward and backward directions to obtain the global state space feature representation.
3. The event flow sign language translation method based on state space according to claim 2 is characterized in that: The method used to divide the event stream data is: the event source data stream is a four-tuple {X, Y, T, P}, where X and Y represent the spatial position coordinates, T represents time, and P represents the polarity of the event. The spatial position with an event is represented as 1, and the spatial position without an event is marked as 0. It is divided according to time t to form segments of size X×Y×t, with a total of T / t segments.
4. The event flow sign language translation method based on state space according to claim 3 is characterized in that: In step S14, the local-global visual feature representation method is: the local spatial feature representation F from the CNN module c Input into the Mamba state space module, use the C2M module to convert the feature map into a feature sequence, pass through a normalization layer, and then use two linear layers to project it into features x and z; use a 1D convolutional layer and SiLU activation function to obtain x for , and x back ; The forward SSM and backward SSM branches are used for bidirectional context perception to generate the forward state space SSM forward and the backward state space SSM backward ; After that, the feature z is activated by SiLU function and SSM forward Features and SSM backward The features are added together after inner product operation and input into the projection layer to obtain the global state space feature representation F M ; Finally, the global state space feature representation F M After the feature sequence is converted into a feature map through the M2C module, it is combined with the local spatial feature representation F C Spliced together, they are input into the next layer of CNN module and Mamba state space module.
5. The event flow sign language translation method based on state space according to claim 4 is characterized in that: The step S14 adopts formula (1), formula (2), and formula (3): x=Linear(Norm(F c )), z=Linear(Norm(F c )) Formula (1) x for =SiLU(Conv(x)), x back =SiLU(Conv(x)) Formula (2) F M =Linear(SSM forward (x for )⊙SiLU(z)+SSM backward (x back )⊙SiLU(z)) Formula (3).
6. The event flow sign language translation method based on state space according to claim 5, characterized in that: Step S4 specifically includes the following steps: S41, the prompt module constructs a prompt sentence for each event stream video, and inputs the prompt sentence and the aligned features into the LLM language module; S42, LLM language module is the LLM language decoder, which uses the cross entropy loss function to calculate the distance between the true sign language sentence and the generated sign language sentence, and generates the final sign language sentence for each event stream video.
7. The event flow sign language translation method based on state space according to claim 6, characterized in that: Step S42 uses formula (4) and formula (5): S={p( <bos>), p(w1), p(w2),..., p(w u ),p( <eos> )} Formula (4)< / eos> < / bos> in, <bos>Indicates the beginning of a sentence. <eos>Indicates the end of the sentence, p(w) represents the probability distribution of the generated words, U represents the number of words in the sign language sentence, S represents the sign sentence Sentence, L represents the loss function Loss, u represents the subscript of the number of sign language words, s u Represents the sentence consisting of the current u words, w1, w2, w u They represent the first, second, and uth word in the sentence respectively, and p represents the conditional probability.< / eos> < / bos>