Isolated sign word recognition method based on event stream and image sequence feature fusion network

By fusing event stream and image sequence features into a network, the limitations of illumination and temporal feature extraction are addressed, improving the accuracy of isolated sign language word recognition and enabling efficient recognition in harsh environments.

CN117894080BActive Publication Date: 2026-11-10UNIV OF SCI & TECH OF CHINA
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410120644.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-29
Publication Date
2026-11-10
Estimated Expiration
2044-01-29

AI Technical Summary

Technical Problem

Existing methods for identifying isolated sign language words based on image sequences are susceptible to the negative effects of harsh ambient lighting conditions such as excessive darkness or overexposure. Furthermore, traditional convolutional neural networks have limitations in extracting temporal features and struggle to capture long-term contextual relationships.

Method used

A method based on feature fusion network of event stream and image sequence is designed. It extracts features from event stream and image sequence simultaneously through deep neural network and performs feature fusion. It takes advantage of the high dynamic range of event camera to compensate for the influence of lighting, while capturing long-term contextual relationships.

Benefits of technology

It improves the accuracy of isolated sign language word recognition, fully utilizes the complementary characteristics of event streams and image sequences, and enhances recognition capabilities in harsh environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117894080B_ABST
    Figure CN117894080B_ABST
Patent Text Reader

Abstract

This invention relates to an isolated sign language word recognition method based on an event stream and image sequence feature fusion network. Compared to traditional image sequence-based isolated sign language word recognition methods, this invention can effectively extract and fuse features from both event streams and image sequences to improve the accuracy of isolated sign language word recognition. The feature fusion network involved in this invention specifically includes a spatial domain feature extraction module and a temporal modeling module. The spatial domain feature extraction module is responsible for encoding the event stream and image sequence separately, fusing features using an attention mechanism, and then performing deep extraction on the fused features. The temporal modeling module is responsible for learning long-term contextual dependencies in the time dimension. This invention can combine the complementary relationship between event stream and image sequence features for spatial domain feature extraction, while effectively modeling temporal features, resulting in a higher recognition accuracy than traditional image sequence-based isolated sign language word recognition methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to an isolated sign language word recognition method based on an event stream and image sequence feature fusion network. Background Technology

[0002] For the deaf and mute community, sign language is an important means of communication, serving as their primary channel for expressing thoughts and feelings. In-depth research into isolated sign language word recognition technology can provide deaf and mute individuals with more convenient and efficient communication tools, enabling them to better participate in social activities and integrate into society. With the continuous development of technology, human-computer interaction has become an indispensable part of social life and work. The development of isolated sign language word recognition technology will also contribute to the advancement of human-computer interaction, providing more natural and intelligent ways of interaction and expanding the ways people communicate with intelligent systems.

[0003] With the advancement of machine learning technologies, represented by deep neural networks, vision-based methods for identifying isolated sign language words have developed rapidly. Traditional methods use video image sequence data, and early methods utilized manually designed features for classification and recognition. Since the 21st century, with the advent of 2D convolutional neural networks, represented by ResNet, deep learning-based methods have achieved remarkable results. 2D convolutional neural networks excel at extracting spatial features from image data; to extract temporal features, methods based on 3D convolutional neural networks have been proposed. However, convolutional neural networks have limitations in extracting temporal features, struggling to capture long-term contextual relationships. Furthermore, the quality of image data is easily affected by harsh ambient lighting conditions such as excessive darkness or overexposure, thus hindering the recognition of isolated sign language words.

[0004] In 2008, the first commercially available event camera was launched. It is a novel bio-neuromorphic visual sensor that captures dynamic changes in a scene based on an event-driven approach. Unlike traditional cameras, it does not output a sequence of image frames. Instead, it generates a series of pixel-level outputs (i.e., an event stream) when changes occur in the real-world scene, recording microsecond-level brightness changes. Event cameras offer advantages such as high dynamic range, high temporal resolution, low latency, and low power consumption, and have been extensively researched in fields such as image restoration, video reconstruction, and robot motion control.

[0005] Isolated sign language words are primarily represented by upper limb movements, exhibiting significant dynamic variations. Event cameras can capture the details of upper limb movements with high temporal resolution. Furthermore, their high dynamic range makes the event stream output less susceptible to harsh ambient lighting conditions, effectively compensating for the limitations of image sequences. Meanwhile, image sequences contain color, texture, and other information neglected in the event stream, making the two complementary. Therefore, it is necessary to design a novel isolated sign language word recognition method based on event streams and image sequences. Summary of the Invention

[0006] To address the issue that isolated sign language word recognition methods based on image sequences are easily affected by harsh ambient lighting conditions such as excessive darkness or overexposure, the high dynamic range of event cameras allows the output event stream to be free from these negative effects, thus compensating for the shortcomings of image sequences. However, this approach ignores information such as color and texture contained in image sequences. To solve the problem of how to make the characteristics of the two complement each other, this invention provides an isolated sign language word recognition method based on a feature fusion network of event stream and image sequence. A deep neural network is designed to simultaneously extract features from the event stream and image sequence and perform feature fusion. Furthermore, it can uncover the long-term contextual relationships of the fused features in the time dimension, thereby improving the accuracy of isolated sign language word recognition.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a method for identifying isolated sign language words based on an event stream and image sequence feature fusion network, comprising the following steps:

[0009] Step 1: Slice the isolated sign language word event stream according to the timestamp of the image sequence to obtain multiple event stream slices. Sample the same number of frames for the image sequence and the event stream slices, and represent the event stream slices in the form of voxel grid to obtain event voxels. Input the sampled image sequence and event voxels into the spatial domain feature extraction module.

[0010] Step 2: The spatial feature extraction module includes an event encoder, an image encoder, an attention fusion unit, and a deep feature extraction unit. The event encoder performs shallow feature extraction on the input event voxels, and the image encoder performs shallow feature extraction on the input image sequence. The two have the same structure, obtaining shallow event features and shallow image features respectively. The event and image shallow features are simultaneously input into the attention fusion unit to capture the spatial correlation of the shallow features, obtaining fused features. The fused features are then sent to the deep feature extraction unit to obtain deep features, which are then input into the temporal modeling module.

[0011] Step 3: The temporal modeling module includes a feature projection unit and a temporal encoder. The deep features are expanded in the time dimension and input into the feature projection unit to project them into a word embedding tensor sequence. A classification label is added to the head of the word embedding tensor sequence and position encoding is added to the sequence. Then, the sequence is input into the temporal encoder to learn the long-term dependencies of the context and obtain a classification label with global attention.

[0012] Step 4: Input the classification label with global attention into the classifier to obtain the isolated sign language word recognition result.

[0013] Further, step 1 includes the following steps:

[0014] Step 11: For an isolated sign language word, the event stream and image sequence are denoted as... in This represents a specific image frame, where M is the sequence length and i is the index of that specific image frame. And 1≤i≤M, the frame rate of the image sequence is denoted as f, therefore the timestamp of the image sequence is denoted as... Event flow can be denoted as Where (x) k ,y k ,t k ,p k ) represents a specific event in an isolated sign language word event stream, where k is the event number and p is the event sequence number. k For the polarity of the event, t k x is the event timestamp. k ,y k These are the pixel x and y coordinates, respectively, and N is the total number of events. (From the event stream timestamp array) Find the match Similar event timestamps The expression is as follows:

[0015]

[0016] In the formula, `searchsorted(·)` represents the nearest neighbor search function. Then, according to... The event stream is segmented to obtain event stream slices, denoted as . e i Represents τ i To τ i+1 The collection of all events within a given time period.

[0017] Step 12: Remove the first frame of the image sequence. and Sample T frames to obtain and Next, the event flow slices are represented in the form of voxel meshes. For a certain event stream slice e j Timestamps of all events in t k The linear contraction to [0, B-1] is given by the following formula:

[0018]

[0019] In the formula, B represents the number of time blocks. t represents the event stamp after linear shrinkage. N Represents an event stream slice e j The last event timestamp in the stream, t1, represents the event stream slice e. jThe first event timestamp in the [database / database].

[0020] Next, the event timestamps are discretized in the B sub-time blocks, and the event at each pixel coordinate is accumulated to obtain the event voxel V(b,y,x), as shown in the following formula:

[0021]

[0022] In the formula, where And 0 ≤ b < B, where b represents the channel dimension corresponding to a certain time block, and (y, x) represents the spatial dimension corresponding to the pixel coordinates. For ease of representation, the event voxel will be denoted as B below. At this point, the sampled image sequence is obtained. and event voxels

[0023] Furthermore, step 2 includes the following steps:

[0024] Step 21: The spatial domain feature extraction module uses the standard ResNet-50-2D as the backbone network. The ResNet-50-2D backbone network contains four Bottleneck feature extraction layers, which are split and placed in different sub-units of the spatial domain feature extraction module. The event encoder and image encoder have the same and symmetrical structure, each containing a two-dimensional convolutional layer, a max pooling layer, and the first Bottleneck feature extraction layer, which extracts the sampled event voxels. and image sequences Inputting the event encoder and image encoder respectively, the shallow event features are obtained. and shallow image features They are represented as follows:

[0025]

[0026] In the formula, f Conv f represents a two-dimensional convolutional layer MP Denotes the max pooling layer, f Bot.1 This represents the first Bottleneck feature extraction layer.

[0027] Step 22: Extract shallow features of the event and shallow image features The input attention fusion unit, based on the channel attention mechanism, consists of two steps: compression and activation. First, event voxels are encoded. By inputting a global average pooling layer, ensuring that all dimensions have a size of 1, spatial compression is achieved. This is a compression operation; then, it passes through a fully connected layer, followed by a ReLU activation layer to reduce the channel dimension, and is then fed back into a fully connected layer, followed by a Sigmoid activation layer... Restore the original channel dimension size, denoted as This is the activation operation. Finally, the compressed and activated shallow event features are multiplied by the shallow image features and then added together to output the fused features. Represented as:

[0028]

[0029] In the formula, f GAP f represents the global average pooling layer. FC.1 f FC.2 denoted as a fully connected layer, δ and σ represent the ReLU activation function and the Sigmoid activation function, respectively, and ⊙ represents matrix element multiplication.

[0030] Step 23: Merge features The input deep feature extraction unit, which consists of the last three Bottleneck feature extraction layers, is then used to output the deep features of the spatial domain feature extraction module. Represented as:

[0031]

[0032] In the formula, f Bot.2 f Bot.3 f Bot.4 These represent the last three Bottleneck feature extraction layers.

[0033] Furthermore, step 3 includes the following steps:

[0034] Step 31: Transfer the deep features Expanding in the time dimension And input the feature projection unit, which includes a max pooling layer, a global average pooling layer, and a multilayer perceptron. First, the feature projection unit is input into the feature projection unit. Each element is symmetrically input into a max-pooling layer and a global average pooling layer. The outputs of the two layers are then concatenated and fed into a multilayer perceptron. The multilayer perceptron consists of three fully connected layers. At this point, the deep features are projected by the feature projection unit to obtain a one-dimensional word embedding tensor sequence, which can be represented as:

[0035]

[0036] In the formula, For word embedding tensor sequences, f Concat It's a splicing operation, f MLP It is a multilayer perceptron.

[0037] Step 32: Next, add a learnable classification label O to the head of the word embedding tensor sequence. cls Obtain the sequence Add positional encoding to the sequence By fixing the temporal relationships and then inputting the sequence into a standard Transformer encoder to learn long-term contextual information, the process can be represented as follows:

[0038]

[0039] In the formula, h(·) represents the standard Transformer encoder. This is the output of the Transformer encoder. Classification labels with global attention This serves as the final output of the entire time series modeling module.

[0040] Furthermore, in step 4, the classifier selected is the Softmax classifier, which includes a fully connected layer and a Softmax layer, to classify the labels. Feed it into the Softmax classifier. After passing through the fully connected layer, we obtain a score tensor with a dimension equal to the total number of isolated sign language word categories. This score tensor is then input into the Softmax layer for sorting, and the index corresponding to the maximum score is the isolated sign language word recognition result.

[0041]

[0042] In the formula, f FC f represents a fully connected layer Softmax This represents the Softmax layer, and result is the final recognition result.

[0043] In a second aspect, the present invention provides an electronic device, including a processor and a memory;

[0044] Memory, used to store computer programs;

[0045] The processor is used to execute the computer program stored in the memory, and when executing it, it implements the steps of the isolated sign word recognition method based on the event stream and image sequence feature fusion network described above.

[0046] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described isolated sign word recognition method based on an event stream and image sequence feature fusion network.

[0047] This invention combines the application of deep neural networks in the field of isolated sign language word recognition, and discloses an isolated sign language word recognition method based on an event flow and image sequence feature fusion network.

[0048] Compared with existing technologies, the beneficial effects are as follows:

[0049] (1) The feature fusion network provided by this invention is the first deep neural network that integrates two modalities, event stream and image sequence, to achieve end-to-end isolated sign word recognition.

[0050] (2) The spatial feature extraction module in the feature fusion network provided by the present invention can simultaneously extract the color, texture and other features contained in the image sequence and the rich sign language word motion features contained in the event stream. Based on attention fusion of the two features, the high dynamic range advantage of the event camera is used to make up for the disadvantage of the image sequence being easily affected by harsh ambient lighting. Considering the complementarity of the two features, the feature description of isolated sign language words is fully extracted.

[0051] (3) The temporal modeling module in the feature fusion network provided by the present invention can better extract the long-term contextual relationship of the fused features in the time dimension compared with the temporal modeling method in the traditional deep neural network, and further improve the accuracy of isolated sign language word recognition. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings and tables used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings and tables described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an overall framework diagram of an isolated sign language word recognition method based on event stream and image sequence feature fusion network involved in this invention;

[0054] Figure 2 This is a flowchart of the event stream and image sequence preprocessing of the present invention;

[0055] Figure 3 This is a detailed structural diagram of the spatial feature extraction module of the present invention;

[0056] Figure 4 This is a structural diagram of the timing modeling module of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0058] like Figure 1 As shown, this embodiment of the invention provides an isolated sign language word recognition method based on an event stream and image sequence feature fusion network, comprising the following steps:

[0059] Step 1: Slice the isolated sign language word event stream according to the timestamp of the image sequence to obtain multiple event stream slices. Sample the same number of frames for the image sequence and the event stream slices, and represent the event stream slices in the form of voxel grid to obtain event voxels. Input the sampled image sequence and event voxels into the spatial domain feature extraction module.

[0060] like Figure 2 As shown, the implementation steps for event stream and image sequence preprocessing are as follows:

[0061] Step 11: For an isolated sign language word, the event stream and image sequence are denoted as... in This represents a specific image frame, where M is the sequence length and i is the index of that specific image frame. And 1≤i≤M, the frame rate of the image sequence is denoted as f, therefore the timestamp of the image sequence is denoted as... Event flow can be denoted as Where (x) k ,y k ,t k ,p k ) represents a specific event in an isolated sign language word event stream, where k is the event number and p is the event sequence number. k For the polarity of the event, t k x is the event timestamp. k ,y k These are the pixel x and y coordinates, respectively, and N is the total number of events. (From the event stream timestamp array) Find the match Similar event timestamps The expression is as follows:

[0062]

[0063] In the formula, searchsorted(·) represents the nearest neighbor search function.

[0064] Next, according to The event stream is segmented to obtain event stream slices, denoted as . e i Represents τ i To τ i+1 The collection of all events within a given time period.

[0065] Step 12: Remove the first frame of the image sequence. and Sample T frames to obtain and Next, the event flow slices are represented in the form of voxel meshes. For a certain event stream slice e j Timestamps of all events in t k The linear contraction to [0, B-1] is given by the following formula:

[0066]

[0067] In the formula, B represents the number of time blocks. t represents the event stamp after linear shrinkage. N Represents an event stream slice e j The last event timestamp in the stream, t1, represents the event stream slice e. j The first event timestamp in the [database / database].

[0068] Next, the event timestamps are discretized in the B sub-time blocks, and the event at each pixel coordinate is accumulated to obtain the event voxel V(b,y,x), as shown in the following formula:

[0069]

[0070] In the formula, where And 0 ≤ b < B, where b represents the channel dimension corresponding to a certain time block, and (y, x) represents the spatial dimension corresponding to the pixel coordinates. For ease of representation, the event voxel will be denoted as B below. At this point, the sampled image sequence is obtained. and event voxels

[0071] Step 2: The spatial feature extraction module consists of an event encoder, an image encoder, an attention fusion unit, and a deep feature extraction unit. The event encoder performs shallow feature extraction on the input event voxels, and the image encoder performs shallow feature extraction on the input image sequence. Both have the same structure, yielding shallow event features and shallow image features, respectively. These event and image shallow features are simultaneously input into the attention fusion unit to capture the spatial correlation of the shallow features, resulting in fused features. The fused features are then fed into the deep feature extraction unit to obtain deep features, which are then input into the temporal modeling module.

[0072] like Figure 3As shown, the implementation steps of the spatial domain feature extraction module are as follows:

[0073] Step 21: The spatial domain feature extraction module uses the standard ResNet-50-2D as the backbone network. The ResNet-50-2D backbone network contains four Bottleneck feature extraction layers, which are split and placed in different sub-units of the spatial domain feature extraction module. The event encoder and image encoder have the same and symmetrical structure, each containing a two-dimensional convolutional layer, a max pooling layer, and a Bottleneck.1 feature extraction layer (called the Bottleneck.1 feature extraction layer), which extracts the sampled event voxels. and image sequences Inputting the event encoder and image encoder respectively, the shallow event features are obtained. and shallow image features They are represented as follows:

[0074]

[0075] In the formula, f Conv f represents a two-dimensional convolutional layer MP Denotes the max pooling layer, f Bot.1 This represents the first Bottleneck feature extraction layer.

[0076] Step 22: Extract shallow features of the event and shallow image features The input attention fusion unit, based on the channel attention mechanism, consists of two steps: compression and activation. First, event voxels are encoded. By inputting a global average pooling layer, ensuring that all dimensions have a size of 1, spatial compression is achieved. This is a compression operation; then, it passes through a fully connected layer, followed by a ReLU activation layer to reduce the channel dimension, and is then fed back into a fully connected layer, followed by a Sigmoid activation layer... Restore the original channel dimension size, denoted as This is the activation operation. Finally, the compressed and activated shallow event features are multiplied by the shallow image features and then added together to output the fused features. Represented as:

[0077]

[0078] In the formula, f GAP f represents the global average pooling layer. FC.1 f FC.2 denoted as a fully connected layer, δ and σ represent the ReLU activation function and the Sigmoid activation function, respectively, and ⊙ represents matrix element multiplication.

[0079] Step 23: Merge features The input deep feature extraction unit, which consists of the last three Bottleneck feature extraction layers (Bottleneck.2, Bottleneck.3, and Bottleneck.4), then the spatial domain feature extraction module finally outputs the deep features. Represented as:

[0080]

[0081] In the formula, f Bot.2 f Bot.3 f Bot.4 These represent the Bootleneck.2 feature extraction layer, Bootleneck.3 feature extraction layer, and Bootleneck.4 feature extraction layer, respectively.

[0082] Step 3: The temporal modeling module consists of a feature projection unit and a temporal encoder. The deep features are unfolded in the time dimension and input into the feature projection unit, projecting them into a word embedding tensor sequence. A classification label is added to the head of the word embedding tensor sequence, and positional encoding is added to the sequence. Then, it is input into the temporal encoder to learn long-term contextual dependencies and obtain a classification label with global attention.

[0083] like Figure 4 As shown, the implementation steps of the time series modeling module are as follows:

[0084] Step 31: Transfer the deep features Expanding in the time dimension And input the feature projection unit, which includes a max pooling layer, a global average pooling layer, and a multilayer perceptron. First, the feature projection unit is input into the feature projection unit. Each element is symmetrically input into a max pooling layer and a global average pooling layer, respectively. The outputs of the two layers are then concatenated and fed into a multilayer perceptron, which consists of three fully connected layers. At this point, the deep features are projected by the feature projection unit to obtain a one-dimensional word embedding tensor sequence, which can be represented as:

[0085]

[0086] In the formula, For word embedding tensor sequences, f Concat It's a splicing operation, f MLP It is a multilayer perceptron.

[0087] Step 32: Add a learnable classification tag O to the head of the word embedding tensor sequence. cls Obtain the sequence Add positional encoding to the sequence By fixing the temporal relationships and then inputting the sequence into a standard Transformer (self-attention model) encoder to learn long-term contextual information, the process can be represented as follows:

[0088]

[0089] In the formula, h(·) represents the standard Transformer encoder. This is the output of the Transformer encoder. Classification labels with global attention This serves as the final output of the entire time series modeling module.

[0090] Step 4: The classifier selected is the Softmax classifier. The Softmax classifier includes a fully connected layer and a Softmax layer, which are used to classify the labels. Feed it into the Softmax classifier. After passing through the fully connected layer, a score tensor with a dimension equal to the total number of isolated sign language word categories is obtained. The score tensor is then input into the Softmax layer for sorting, and the index corresponding to the maximum score is the isolated sign language word recognition result.

[0091]

[0092] In the formula, f FC f represents a fully connected layer Softmax This represents the Softmax layer, and result is the final recognition result.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (which may be ROM / RAM, CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (which may be a computer, server, smartphone, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0094] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying isolated sign language words based on an event stream and image sequence feature fusion network, characterized in that, Includes the following steps: Step 1: Slice the isolated sign language word event stream according to the timestamp of the image sequence to obtain multiple event stream slices. Sample the same number of frames for the image sequence and the event stream slices. Represent the event stream slices in the form of voxel grid to obtain event voxels. Input the sampled image sequence and event voxels into the spatial domain feature extraction module. Step 2: The spatial feature extraction module includes an event encoder, an image encoder, an attention fusion unit, and a deep feature extraction unit; The event encoder extracts shallow features from the input event voxels, and the image encoder extracts shallow features from the input image sequence. The two have the same structure, and the event shallow features and image shallow features are obtained respectively. The event and image shallow features are simultaneously input into the attention fusion unit to capture the spatial correlation of the shallow features and obtain the fused features. The fused features are sent to the deep feature extraction unit to obtain the deep features and then input into the temporal modeling module. Step 3: The temporal modeling module includes a feature projection unit and a temporal encoder; the deep features are expanded in the time dimension and input into the feature projection unit to project them into a word embedding tensor sequence; a classification label is added to the head of the word embedding tensor sequence and position encoding is added to the sequence, and then input into the temporal encoder to learn the long-term dependencies of the context and obtain a classification label with global attention; Step 4: Input the classification label with global attention into the classifier to obtain the isolated sign language word recognition result.

2. The isolated sign language word recognition method based on an event stream and image sequence feature fusion network according to claim 1, characterized in that, Step 1 includes: Step 11: For an isolated sign language word, the event stream and image sequence are denoted as... ,in This represents a specific frame of an image. For sequence length, This is the index of a specific frame of an image. and The frame rate of the image sequence is denoted as Therefore, the timestamp of the image sequence is recorded as Event log is ,in This indicates an isolated event within a sign language event stream. For the event sequence number, For the polarity of the event, For event timestamps, These are the x and y coordinates of pixels, respectively. The total number of events; From the event stream timestamp array Find the match Similar event timestamps The expression is as follows: (1) In the formula, This represents the nearest neighbor search function; according to The event stream is segmented to obtain event stream slices, denoted as . , express to The collection of all events within a given time period; Step 12: Remove the first frame of the image sequence. and sampling Frame, obtain and Then, the event flow slices are characterized in the form of voxel meshes. ; For a certain event stream slice ,Will Timestamps of all events linear shrinkage to The formula is as follows: (2) In the formula, The number of time blocks. Represents the event stamp after linear shrinkage. Represents an event stream slice The last event timestamp in the data. Represents an event stream slice The first event timestamp in; exist The event timestamps are discretized within each sub-time block, and the event at each pixel coordinate is accumulated to obtain an event voxel. The formula is as follows: (3) In the formula, where and , This indicates that a certain sub-time block corresponds to the channel dimension. This indicates that pixel coordinates correspond to spatial dimensions, and event voxels are denoted as... ; At this point, the sampled image sequence is obtained. and event voxels .

3. The isolated sign language word recognition method based on an event stream and image sequence feature fusion network according to claim 2, characterized in that, Step 2 includes: Step 21: The spatial domain feature extraction module uses the standard ResNet-50-2D as the backbone network. The ResNet-50-2D backbone network contains four Bottleneck feature extraction layers, which are split and placed in different sub-units of the spatial domain feature extraction module. The event encoder and image encoder have the same and symmetrical structure, each containing a two-dimensional convolutional layer, a max pooling layer, and the first Bottleneck feature extraction layer, which are used to extract sampled event voxels. and image sequences Inputting the event encoder and image encoder respectively, the shallow event features are obtained. and shallow image features They are represented as follows: (4) In the formula, Represents a two-dimensional convolutional layer. Indicates the max pooling layer. This represents the first Bottleneck feature extraction layer; Step 22: Extract shallow features of the event and shallow image features The input attention fusion unit, based on the channel attention mechanism, consists of two steps: compression and activation. First, event voxels are encoded. By inputting a global average pooling layer, ensuring that all dimensions have a size of 1, spatial compression is achieved. This is the compression operation; then, it passes through a fully connected layer, followed by a ReLU activation layer to reduce the channel dimension, and then is fed back into a fully connected layer, followed by a Sigmoid activation layer to... Restore the original channel dimension size, denoted as This is the activation operation; finally, the compressed and activated shallow event features are multiplied by the shallow image features and then added together to output the fused features. Represented as: (5) In the formula, This indicates a global average pooling layer. , Indicates a fully connected layer. , These represent the ReLU activation function and the Sigmoid activation function, respectively. This refers to element-wise matrix multiplication. Step 23: Merge features The input deep feature extraction unit, which includes the last three Bottleneck feature extraction layers, is then used to output the deep features of the spatial domain feature extraction module. Represented as: (6) In the formula, , , These represent the last three Bottleneck feature extraction layers.

4. The isolated sign language word recognition method based on an event stream and image sequence feature fusion network according to claim 3, characterized in that, Step 3 includes: Step 31: Transfer the deep features Expanding in the time dimension And input the feature projection unit, which includes a max pooling layer, a global average pooling layer, and a multilayer perceptron. First, the feature projection unit is input into the feature projection unit. Each element is symmetrically input into a max-pooling layer and a global average pooling layer. The outputs of the two layers are then concatenated and fed into a multilayer perceptron. The multilayer perceptron consists of three fully connected layers. At this point, the deep features are projected by the feature projection unit to obtain a one-dimensional word embedding tensor sequence, represented as: (7) In the formula, Embed tensor sequences for words, It's a splicing operation. It is a multilayer perceptron; Step 32: Add a learnable classification tag to the head of the word embedding tensor sequence. Obtain the sequence Add position encoding to the sequence By fixing the temporal relationships, and then inputting the sequence into a standard Transformer encoder to learn long-term contextual information, the process can be represented as follows: (8) In the formula, This represents a standard Transformer encoder. This is the output of the Transformer encoder. Classification labels with global attention This serves as the final output of the entire time series modeling module.

5. The isolated sign language word recognition method based on an event stream and image sequence feature fusion network according to claim 4, characterized in that, In step 4, a Softmax classifier is selected as the classifier. The Softmax classifier includes a fully connected layer and a Softmax layer, which are used to classify the labels. Feed it into the Softmax classifier. After passing through the fully connected layer, we obtain a score tensor with a dimension equal to the total number of isolated sign language word categories. This score tensor is then input into the Softmax layer for sorting, and the index corresponding to the maximum score is the isolated sign language word recognition result. (9) In the formula, Indicates a fully connected layer. This represents the Softmax layer, and result is the final recognition result.

6. An electronic device, characterized in that, Including processor and memory; Memory, used to store computer programs; A processor is configured to execute a computer program stored in memory, which, when executed, implements the isolated sign word recognition method based on an event stream and image sequence feature fusion network as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the isolated sign word recognition method based on the event stream and image sequence feature fusion network as described in any one of claims 1-5.

Citation Information

Patent Citations

  • Gesture recognition method and electronic equipment

    CN115661941A

  • Recognition Actions on Event Based Cameras with Motion Event Features

    US20180218203A1