An isolated sign language word recognition method based on event camera

Abstract

The present invention relates to an isolated sign language word recognition method based on an event camera. The method comprises: using an event camera to re-shoot an isolated sign language word video to obtain an isolated sign language word event stream; processing the event stream slices according to a fixed time window; designing an event-driven sampling algorithm to sample the event stream slices to obtain key frames; representing the key frames in the form of a voxel grid to obtain low-speed and high-speed key frame tensors; and inputting the key frames into a convolutional neural network containing a high-speed and low-speed motion information fusion module and a temporal shift module for classification to achieve isolated sign language word recognition. The present invention, incorporating the characteristics of an event camera, fully extracts key frame information and high-speed motion information from the isolated sign language word event stream, effectively alleviating the problems of traditional camera-based recognition methods, which are limited by video information redundancy and motion blur, thereby improving recognition efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an isolated sign language word recognition method based on an event camera. Background Art

[0002] Isolated word sign language recognition can be viewed as a fine-grained action classification task, designed to help the hearing-impaired overcome communication barriers with the general public. It has significant research and application value. Data acquisition methods can be categorized as vision-based or wearable sensor-based. In recent years, due to the rapid development of deep learning technology and its hardware cost advantages, vision-based methods have become a research hotspot.

[0003] Traditional camera-based recognition methods use isolated sign language word video data. Early methods attempted to extract handcrafted features from this data for further classification and recognition. In recent years, deep learning-based methods have significantly improved recognition performance compared to earlier handcrafted feature-based methods. Deep neural networks, represented by convolutional neural networks, can effectively extract spatiotemporal features from data. In 2015, a method based on 3D convolutional neural networks was proposed, achieving remarkable results (see J. Huang, W. Zhou, H. Li, and W. Li, “Sign language recognition using 3D convolutional neural networks” in Proceedings of the IEEE International Conference on Multimedia and Expo, 2015, pp. 1-6). However, the presence of a large amount of redundant information in the video data, the noisy background of the sign language demonstration environment, and the motion blur of the sign language movements can all negatively impact recognition performance.

[0004] An event camera is a novel bio-neuromorphic visual sensor that outputs data in the form of an event stream. It inherently contains temporal information and has a sparse spatial distribution. It records microsecond-scale pixel-level brightness changes. Each pixel on the sensor outputs independently and asynchronously, offering the advantages of low latency, high dynamic range, high temporal resolution, and low power consumption. Current research on event cameras has covered numerous fields, including robotic control, object detection, and image restoration. In the field of action recognition, the high temporal resolution of event cameras effectively avoids motion blur, recording only brightness changes and thus being sensitive only to motion. This allows them to effectively capture motion information in noisy environments, facilitating recognition. Isolated sign language words have significant motion characteristics, so it is necessary to design a novel isolated sign language word recognition method based on the characteristics of event cameras. Summary of the Invention

[0005] The technical problem solved by the present invention is that a large amount of redundant information exists in the isolated sign language word video data used in the isolated sign language word recognition method based on traditional cameras, and a random sampling strategy is used when sampling video frames. The sampled video frames have similar or invalid information, and video frames containing key information cannot be effectively extracted, resulting in low recognition efficiency. At the same time, due to the low temporal resolution and high latency of traditional cameras, the output video has the problem of motion blur, and the recognition accuracy is poor. The present invention provides an isolated sign language word recognition method based on an event camera, designs an event-driven sampling algorithm to extract key frame information of isolated sign language words, improves recognition efficiency, and utilizes the high temporal resolution characteristics of the event camera to design a high- and low-speed motion information fusion module to fuse high-speed motion information of isolated sign language words, avoiding the influence of motion blur and thus improving recognition accuracy.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides an isolated sign language word recognition method based on an event camera, comprising the following steps:

[0008] Step 1. Obtaining an isolated sign language word event stream: Use an event camera to re-shoot the isolated sign language word video. Before re-shooting, add padding and borders to the video based on the resolution ratio of the isolated sign language word video, the display, and the event camera, then stretch it to full screen. The relative position of the event camera and the display is fixed. During re-shooting, the isolated sign language word video is played at the original frame rate. The event camera outputs a cluster of events with positive and negative polarity in a time sequence and stores them in an array to obtain the isolated sign language word event stream.

[0009] Step 2: Generate low-speed and high-speed keyframe tensors: Slice the isolated sign language word event stream obtained in step 1 according to a fixed time window length to obtain multiple event stream slices, record the sequence number of the last event in each event stream slice in the isolated sign language word event stream to construct a sequence number array, count the number of events in each event stream slice to construct an event number array, sample the event stream slices using an event-driven sampling algorithm based on the event number array to obtain a set of keyframes, locate all events in each keyframe based on the sequence number array, and represent the set of keyframes in the form of a voxel grid using all events in each keyframe. During the representation, set a variable time block parameter. Setting the time block parameter indicates that the event distribution in a set of keyframes is counted at a low time resolution to obtain a low-speed keyframe tensor, and setting the time block parameter indicates that the event distribution in the same set of keyframes is counted at a high time resolution to obtain a high-speed keyframe tensor.

[0010] Step 3. Use a convolutional neural network classification with a high- and low-speed motion information fusion module and a timing shift module to obtain the recognition result: first, pre-process the low-speed key frame tensor and the high-speed key frame tensor obtained in step 2 and input them into the high- and low-speed motion information fusion module to obtain a fused tensor; then input the fused tensor into the timing shift module to obtain a feature tensor, and finally send the feature tensor into the Softmax classifier to obtain the isolated sign language word recognition result.

[0011] Furthermore, obtaining the isolated sign language word event stream in step 1 includes the following steps:

[0012] Step 11: Before reshooting, use the OPENCV image processing tool library to add padding and borders around each frame of the isolated sign language word video. The calculation formulas for the padding length h along the height direction and the padding length w along the width direction of the isolated sign language word video are as follows:

[0013]

[0014]

[0015] Where H v and W v are the height and width of the isolated sign language word video resolution, H d and W d where ∫ is the height and width of the display resolution, and b is the border width. The isolated sign language video is then proportionally stretched to full screen until the height is the same as the display resolution, and offset by a certain length along the width to center the video on the display. The offset length is calculated as follows:

[0016]

[0017] Where H c and W c are the height and width of the event camera resolution respectively; the relative positions of the display and the event camera are fixed so that the boundary of the event camera imaging screen coincides with the border added to the isolated sign language word video;

[0018] When reshooting, play the isolated word sign language video at the original frame rate, open the event camera, and store a cluster of events with positive and negative polarity output by the event camera in the form of an array to obtain the isolated sign language word event stream. Where (x k ,y k ,t k ,p k ) represents an event in the isolated sign language word event stream, k is the event sequence number, p k is the polarity of the event, t k is the timestamp, x k ,y kare the pixel horizontal and vertical coordinates respectively, and N is the total number of events.

[0019] Furthermore, obtaining the sequence number array and the event number array in step 2 includes the following steps:

[0020] Step 21: Set a fixed time window length Length = 1 / P, where P is the frame rate of the isolated sign language word video. Slice the isolated sign language word event stream according to Length to obtain multiple event stream slices. e i Slice a certain event stream; record the last event of each event stream slice in the isolated sign language word event stream as l i , get the sequence number array The number of events recorded in each event stream slice is recorded as E i , get the event number array Where M is the total number of event stream slices, i is the index subscript of a certain event stream slice, i is an integer and 1≤i≤M.

[0021] Furthermore, the event-driven sampling algorithm in step 2 includes the following steps:

[0022] Step 22, acquiring key frames based on the event-driven sampling algorithm includes two parts: calculating the cumulative integral function of the event number array and sampling key frames according to the nearest neighbor principle;

[0023] Count event count array The cumulative integral function D m , the formula is as follows:

[0024]

[0025] Where m is the index subscript of the event array element, which corresponds one-to-one to the event stream slice index subscript, and μ is a hyperparameter;

[0026] According to the nearest neighbor principle, key frames are sampled, D m ∈(0,1], let the number of sampling frames be T, and divide the interval (0,1] into T segments equally Select a random value in each subinterval and find the cumulative integral function D of the event number array closest to the random value. m The event number array element index subscript corresponding to the function value is marked as the key frame index subscript f τ , the formula is as follows:

[0027]

[0028] Where, ( is a subinterval, τ is an integer and 1≤τ≤T, rand is a random value generation function, nearest is the nearest neighbor function, and samples are taken in each subinterval in turn. The event stream slice is indexed by the key frame index subscript to obtain a set of key frames recorded as in For a key frame;

[0029] Furthermore, obtaining the low-speed and high-speed key frame tensors in step 2 includes the following steps:

[0030] Step 23, according to the sequence number array and a set of keyframes A key frame All events in are represented as in, Keyframe The last event number in The last event number of the previous keyframe, representing the keyframe online in the form of a voxel grid First, the keyframe The timestamps of all events in k Linear shrinkage to [0, B-1], the formula is as follows:

[0031]

[0032] Where B is the time block parameter, represents the event stamp after linear shrinkage, Represents a keyframe The last event timestamp in Indicates the last event timestamp of the key frame before this key frame; then the key frame tensor V(b,y) in the form of a voxel grid is generated k ,x k ), the formula is as follows:

[0033]

[0034] Where b is an integer and b∈[0,B-1], k is the event number, p k is the event polarity, x k ,y k They are the horizontal and vertical coordinates of the pixel respectively, max represents the maximum value function, and the size of the generated key frame tensor is 2B×H×W. The first dimension 2B represents the channel dimension size, the coefficient 2 of 2B represents the positive and negative event polarity, B is the time block parameter, H and W represent the height and width of the spatial dimension respectively; for a set of key frames with a sampling frame number of T Setting the time block parameter B = 1 means that the event distribution in a set of key frames is counted at a low time resolution to obtain a low-speed key frame tensor V of size T×2×H×W.L Setting the time block parameter B = 4 means that the event distribution in a set of key frames is counted at a high time resolution to obtain a high-speed key frame tensor V of size T×8×H×W H .

[0035] Furthermore, the step 3 includes using the high-speed and low-speed motion information fusion module to obtain a fusion tensor, which includes the following steps:

[0036] Step 31: The low-speed and high-speed key frame tensors V obtained in step 2 are L 、V H The two-dimensional convolution layer and the maximum pooling layer are respectively used to achieve dimensionality increase in the channel dimension and dimensionality reduction in the spatial dimension, and the low-speed key frame tensor U with a size of T×64×H / 4×W / 4 is obtained. L and the preprocessed high-speed keyframe tensor U with size T×256×H / 4×W / 4 H :

[0037] U L ,U H =Maxpool(Conv(V L )),Maxpool(Conv(V H )) (8)

[0038] In the formula, Conv represents the two-dimensional convolution layer, Maxpool represents the maximum pooling layer;

[0039] Will U L 、U H The high-speed and low-speed motion information fusion module is firstly used to fusion the pre-processed high-speed key frame tensor U H Perform a global average pooling operation in the spatial domain to make the size of the spatial dimension 1, achieve spatial compression, and obtain the compressed high-speed key frame tensor Then It is sent to the fully connected layer, and the channel dimension is reduced by the ReLU activation layer, and then sent to the fully connected layer and the Sigmoid activation layer to finally make Channel dimension and U L Consistent, get the high-speed key frame tensor after excitation Finally, the high-speed key frame tensor after excitation and the preprocessed low-speed key frame tensor U L Perform matrix multiplication and then add it to the preprocessed low-speed key frame tensor U L Add together to get the fused tensor U of size T×64×H / 4×W / 4 F :

[0040]

[0041]

[0042]

[0043] Where GAP represents global average pooling, W1 and W2 represent fully connected layers, δ and σ represent ReLU activation function and Sigmoid activation function respectively, and ⊙ represents matrix element multiplication.

[0044] Furthermore, in the above-mentioned isolated sign language word recognition method based on event camera, the step 3 of using the temporal shift module to obtain the feature tensor and using the Softmax classifier to obtain the isolated sign language word recognition result includes the following steps:

[0045] Step 32: Fuse the fused tensor U L Input the timing shift module to obtain the feature tensor. The timing shift module consists of two parts: timing shift operation and backbone network.

[0046] The standard Resnet-50-2D backbone network is selected as the backbone network. The Resnet-50-2D backbone network contains four Bottleneck feature extraction subnetworks, each of which extracts features from the fused tensor in different spatial dimensions. The temporal shift operation shifts the fused tensor input to each Bottleneck feature extraction subnetwork in the time dimension, that is, the temporal shift operation is embedded in the heads of the four Bottleneck feature extraction subnetworks. Assuming the number of channels contained in the fused tensor input to a Bottleneck feature extraction subnetwork is in_channels and the shift depth is d, the number of channels c shifted in one direction is:

[0047] c=in_channels / d (12)

[0048] The fused tensor is split into three parts in the channel dimension: the left end, the middle end, and the right end. The left end contains the first c channels, the middle end contains the cth to 2cth channels, and the remaining channels are the right end. The left end is shifted one unit to the left in the time dimension, and the middle end is shifted one unit to the right in the time dimension. The vacant dimension is filled with zeros to establish a bidirectional temporal relationship. The right end remains unchanged to maintain the stability of its own spatial structure. The formula is as follows:

[0049]

[0050] Where, Represents the left, middle, and right ends of the split fusion tensor. Lshift and Rshift represent left and right shift operations respectively. Concat represents reconnecting the split fusion tensor to obtain the fusion tensor to be input into a Bottleneck feature extraction subnetwork after time sequence shifting.

[0051] Fused Tensor U F After four Bottleneck feature extraction sub-networks embedded with temporal shift operations, the output sizes become T×256×H / 4×W / 4, T×512×H / 8×W / 8, T×512×H / 16×W / 16, and T×2048×H / 32×W / 32 respectively. The output of the last Bottleneck feature extraction sub-network is the feature tensor output by the temporal shift module, which is recorded as

[0052] Finally, the feature tensor The feature tensor is fed into the Softmax classifier, which consists of an average pooling layer, a linear fully connected layer, and a Softmax layer. After the average pooling layer and the linear fully connected layer, the size of the feature tensor becomes T × classes. The class dimension size corresponds to the total number of isolated sign language word categories, and the value is the score of each category. Then, the T scores of each category are averaged to return the feature tensor size to classes. After the final Softmax layer sorts the scores, the index corresponding to the maximum score is output as the isolated sign language word recognition result:

[0053]

[0054] In the formula, Avgpool is the average pooling layer, Linear is the linear fully connected layer, Average is the average operation, and Softmax is the Softmax layer.

[0055] In a second aspect, the present invention provides an electronic device comprising a processor and a memory;

[0056] Memory for storing computer programs;

[0057] The processor is used to execute the computer program stored in the memory, and implements the above-mentioned isolated sign language word recognition method based on event camera during execution.

[0058] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned isolated sign language word recognition method based on an event camera.

[0059] This paper, combining the application of convolutional neural networks in sign language recognition, discloses a method for isolated sign language word recognition based on an event camera. By leveraging the advantages of event cameras and fully exploiting the output characteristics of event streams, the paper offers the following innovations compared to existing technologies:

[0060] (1) The present invention applies the event camera to the isolated sign language word recognition task for the first time. The event camera outputs an event stream. Traditional methods cannot use this data type for recognition. The present invention can use the isolated sign language word event stream as input to complete the isolated sign language word recognition task, thus expanding the application scenarios of the event camera.

[0061] (2) The present invention combines the sparsity characteristics of event stream data output by event cameras and designs an event-driven sampling algorithm to efficiently sample event stream slices containing rich information as key frames, reducing the use of redundant frames and improving recognition efficiency.

[0062] (3) The present invention combines the high temporal resolution advantage of the event camera, represents the key frames sampled from the event stream slices online in the form of a voxel grid, sets the time block parameters to represent the low-speed key frame tensor obtained by the low temporal resolution representation, and sets the time block parameters to represent the high-speed key frame tensor obtained by the high temporal resolution representation. In the convolutional network used for classification and recognition, a high- and low-speed motion information fusion module is designed to fuse the low-speed and high-speed key frame tensors to obtain motion detail information at high temporal resolution that is difficult to use by recognition methods based on traditional cameras. A temporal shift module is designed to quickly extract temporal information, thereby improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings and tables required for describing the embodiments. Obviously, the drawings and tables described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a flow chart of the overall solution of the present invention;

[0065] Figure 2 This is a schematic diagram of the event flow for obtaining isolated sign language words in the present invention;

[0066] Figure 3 This is a flow chart for implementing the event-driven sampling algorithm in the present invention;

[0067] Figure 4 The specific structure of the convolutional neural network containing the high-speed and low-speed motion information fusion module and the timing shift module in the present invention;

[0068] Figure 5 The recognition effect of the present invention is compared with that of multiple traditional camera-based isolated sign language word recognition methods on the WLASL2000 dataset. DETAILED DESCRIPTION

[0069] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0070] like Figure 1 As shown, an embodiment of the present invention provides a flowchart of an overall solution for isolated sign language word recognition based on an event camera.

[0071] The main steps are as follows:

[0072] Step 1. Obtaining an isolated sign language word event stream: Use an event camera to re-shoot the isolated sign language word video. Before re-shooting, add padding and borders to the video based on the resolution ratio of the isolated sign language word video, the display, and the event camera, then stretch it to full screen. The relative position of the event camera and the display is fixed. During re-shooting, the isolated sign language word video is played at the original frame rate. The event camera outputs a cluster of events with positive and negative polarity in a time sequence and stores them in an array to obtain the isolated sign language word event stream.

[0073] like Figure 2 As shown in the figure, the schematic diagram of the event flow for obtaining isolated sign language words is reproduced, and the implementation steps are as follows:

[0074] Step 11: Before reshooting, use the OPENCV image processing tool library to add black padding and red borders to the isolated sign language word video. The calculation formulas for the padding length h along the height direction and the padding length w along the width direction of the isolated sign language word video are as follows:

[0075]

[0076]

[0077] Where H v and W v are the height and width of the isolated sign language word video resolution, H d and W d where _ is the height and width of the display resolution, respectively; b is the border width, set to 2; then the isolated sign language video is proportionally stretched to full screen until the height is the same as the display resolution, and offset along the width by a certain length so that the video image is centered on the display. The offset length is calculated as follows:

[0078]

[0079] Where H c and W care the height and width of the event camera resolution, respectively; the relative positions of the display and the event camera are fixed so that the boundary of the event camera imaging screen coincides with the red border added to the isolated sign language word video;

[0080] When reshooting, play the isolated word sign language video at the original frame rate, open the event camera, and store a cluster of events with positive and negative polarity output by the event camera in the form of an array to obtain the isolated sign language word event stream. Where (x k ,y k ,t k ,p k ) represents an event in the isolated sign language word event stream, k is the event sequence number, p k is the polarity of the event, t k is the timestamp, x k ,y k are the pixel horizontal and vertical coordinates respectively, and N is the total number of events.

[0081] Step 2: Generate low-speed and high-speed keyframe tensors: Slice the isolated sign language word event stream obtained in step 1 according to a fixed time window length to obtain multiple event stream slices, record the sequence number of the last event in each event stream slice in the isolated sign language word event stream to construct a sequence number array, count the number of events in each event stream slice to construct an event number array, sample the event stream slices using an event-driven sampling algorithm based on the event number array to obtain a set of keyframes, locate all events in each keyframe based on the sequence number array, and represent the set of keyframes in the form of a voxel grid using all events in each keyframe. During the representation, set a variable time block parameter. Setting the time block parameter indicates that the event distribution in a set of keyframes is counted at a low time resolution to obtain a low-speed keyframe tensor, and setting the time block parameter indicates that the event distribution in the same set of keyframes is counted at a high time resolution to obtain a high-speed keyframe tensor.

[0082] Step 21: Set a fixed time window length Length = 1 / P, where P is the frame rate of the isolated sign language word video. Slice the isolated sign language word event stream according to Length to obtain multiple event stream slices. e i Slice a certain event stream; record the last event of each event stream slice in the isolated sign language word event stream as l i , get the sequence number array The number of events recorded in each event stream slice is recorded as E i , get the event number array Where M is the total number of event stream slices, i is the index subscript of a certain event stream slice, i is an integer and 1≤i≤M;

[0083] like Figure 3As shown in the figure, the event-driven sampling algorithm implementation flow chart is as follows:

[0084] Step 22, acquiring key frames based on the event-driven sampling algorithm includes two parts: calculating the cumulative integral function of the event number array and sampling key frames according to the nearest neighbor principle;

[0085] Count event count array The cumulative integral function D m , the formula is as follows:

[0086]

[0087] Where m is the index subscript of the event number array element, which corresponds one-to-one to the event stream slice index subscript, and μ is a hyperparameter set to 0.5;

[0088] According to the nearest neighbor principle, key frames are sampled, D m ∈(0,1], let the number of sampling frames be T, and divide the interval (0,1] into T segments equally Select a random value in each subinterval and find the cumulative integral function D of the event number array closest to the random value. m The event number array element index subscript corresponding to the function value is marked as the key frame index subscript f τ , the formula is as follows:

[0089]

[0090] Where, is a subinterval, τ is an integer and 1≤τ≤T, rand is a random value generation function, nearest is the nearest neighbor function, and samples are taken in each subinterval in turn. The event stream slice is indexed by the key frame index subscript to obtain a set of key frames recorded as in For a key frame;

[0091] Step 23, according to the sequence number array and a set of keyframes A key frame All events in are represented as in, Keyframe The last event number in The last event sequence number of the key frame before this key frame, representing the key frame online in the form of a voxel grid First, the keyframe The timestamps of all events in k Linear shrinkage to [0, B-1], the formula is as follows:

[0092]

[0093] Where B is the time block parameter, represents the event stamp after linear shrinkage, Represents a keyframe The last event timestamp in Indicates the last event timestamp of the key frame before this key frame; then the key frame tensor V(b,y) in the form of a voxel grid is generated k ,x k ), the formula is as follows:

[0094]

[0095] Where b is an integer and b∈[0,B-1], k is the event number, p k is the event polarity, x k ,y k They are the horizontal and vertical coordinates of the pixel respectively, max represents the maximum value function, and the size of the generated key frame tensor is 2B×H×W. The first dimension 2B represents the channel dimension size, the coefficient 2 of 2B represents the positive and negative event polarity, B is the time block parameter, H and W represent the height and width of the spatial dimension respectively; for a set of key frames with a sampling frame number of T Setting the time block parameter B = 1 means that the event distribution in a set of key frames is counted at a low time resolution to obtain a low-speed key frame tensor V of size T×2×H×W. L Setting the time block parameter B = 4 means that the event distribution in a set of key frames is counted at a high time resolution to obtain a high-speed key frame tensor V of size T×8×H×W H Temporal resolution refers to the time interval of a certain size used to statistically analyze the distribution of events. Low temporal resolution corresponds to a time interval equal to the inverse of the frame rate of the isolated sign language word video, while high temporal resolution corresponds to a time interval less than the inverse of the frame rate of the isolated sign language word video.

[0096] Step 3. Use a convolutional neural network classification with a high- and low-speed motion information fusion module and a timing shift module to obtain the recognition result: first, pre-process the low-speed key frame tensor and the high-speed key frame tensor obtained in step 2 and input them into the high- and low-speed motion information fusion module to obtain a fused tensor; then input the fused tensor into the timing shift module to obtain a feature tensor, and finally send the feature tensor into the Softmax classifier to obtain the isolated sign language word recognition result.

[0097] like Figure 4 As shown in the figure, the specific structure of the convolutional neural network containing the high-speed and low-speed motion information fusion module and the timing shift module is implemented as follows:

[0098] Step 31: The low-speed and high-speed key frame tensors V obtained in step 2 are L 、VH The two-dimensional convolution layer and the maximum pooling layer are respectively used to achieve dimensionality increase in the channel dimension and dimensionality reduction in the spatial dimension, and the low-speed key frame tensor U with a size of T×64×H / 4×W / 4 is obtained. L and the preprocessed high-speed keyframe tensor U with size T×256×H / 4×W / 4 H :

[0099] U L ,U H =Maxpool(Conv(V L )),Maxpool(Conv(V H )) (8)

[0100] In the formula, Conv represents the two-dimensional convolution layer, Maxpool represents the maximum pooling layer;

[0101] Will U L 、U H The high-speed and low-speed motion information fusion module is firstly used to fusion the pre-processed high-speed key frame tensor U H Perform a global average pooling operation in the spatial domain to make the size of the spatial dimension 1, achieve spatial compression, and obtain the compressed high-speed key frame tensor Then It is sent to the fully connected layer, and the channel dimension is reduced by the ReLU activation layer, and then sent to the fully connected layer and the Sigmoid activation layer to finally make Channel dimension and U L Consistent, get the high-speed key frame tensor after excitation Finally, the high-speed key frame tensor after excitation and the preprocessed low-speed key frame tensor U L Perform matrix multiplication and then add it to the preprocessed low-speed key frame tensor U L Add together to get the fused tensor U of size T×64×H / 4×W / 4 F :

[0102]

[0103]

[0104]

[0105] Where GAP represents global average pooling, W1 and W2 represent fully connected layers, δ and σ represent ReLU activation function and Sigmoid activation function respectively, and ⊙ represents matrix element multiplication.

[0106] Furthermore, in the above-mentioned isolated sign language word recognition method based on event camera, the step 3 of using the temporal shift module to obtain the feature tensor and using the Softmax classifier to obtain the isolated sign language word recognition result includes the following steps:

[0107] Step 32: Fuse the fused tensor U L Input the timing shift module to obtain the feature tensor. The timing shift module consists of two parts: timing shift operation and backbone network.

[0108] The standard Resnet-50-2D backbone network is selected as the backbone network. The Resnet-50-2D backbone network contains four Bottleneck feature extraction subnetworks, each of which extracts features from the fused tensor in different spatial dimensions. The temporal shift operation shifts the fused tensor input to each Bottleneck feature extraction subnetwork in the time dimension, that is, the temporal shift operation is embedded in the heads of the four Bottleneck feature extraction subnetworks. Assuming the number of channels contained in the fused tensor input to a Bottleneck feature extraction subnetwork is in_channels and the shift depth is d, the number of channels c shifted in one direction is:

[0109] c=in_channels / d (12)

[0110] The fused tensor is split into three parts in the channel dimension: the left end, the middle end, and the right end. The left end contains the first c channels, the middle end contains the cth to 2cth channels, and the remaining channels are the right end. The left end is shifted one unit to the left in the time dimension, and the middle end is shifted one unit to the right in the time dimension. The vacant dimension is filled with zeros to establish a bidirectional temporal relationship. The right end remains unchanged to maintain the stability of its own spatial structure. The formula is as follows:

[0111]

[0112] Where, Represents the left, middle, and right ends of the split fusion tensor. Lshift and Rshift represent left and right shift operations respectively. Concat represents reconnecting the split fusion tensor to obtain the fusion tensor to be input into a Bottleneck feature extraction subnetwork after time sequence shifting.

[0113] Fused Tensor U FAfter four Bottleneck feature extraction sub-networks embedded with temporal shift operations, the output sizes become T×256×H / 4×W / 4, T×512×H / 8×W / 8, T×512×H / 16×W / 16, and T×2048×H / 32×W / 32 respectively. The output of the last Bottleneck feature extraction sub-network is the feature tensor output by the temporal shift module, which is recorded as

[0114] Finally, the feature tensor The feature tensor is fed into the Softmax classifier, which consists of an average pooling layer, a linear fully connected layer, and a Softmax layer. After the average pooling layer and the linear fully connected layer, the size of the feature tensor becomes T × classes. The class dimension size corresponds to the total number of isolated sign language word categories, and the value is the score of each category. Then, the T scores of each category are averaged to return the feature tensor size to classes. After the final Softmax layer sorts the scores, the index corresponding to the maximum score is output as the isolated sign language word recognition result:

[0115]

[0116] In the formula, Avgpool is the average pooling layer, Linear is the linear fully connected layer, Average is the average operation, and Softmax is the Softmax layer.

[0117] like Figure 5 As shown in FIG, the recognition effect of an isolated sign language word recognition method based on an event camera mentioned in the present invention is compared with multiple isolated sign language word recognition methods based on traditional cameras on the WLASL2000 dataset:

[0118] The WLASL2000 dataset is an open-source dataset of isolated sign language words, containing 21,095 samples across 2,000 categories. This paper obtains isolated sign language word event streams by reshooting isolated sign language word videos from this dataset. Recognition accuracy and model convergence cycles are used as test metrics. Higher recognition accuracy indicates better recognition results, while lower accuracy indicates worse recognition results. Lower model convergence cycles indicate higher recognition efficiency, while lower accuracy indicates lower recognition efficiency.

[0119] In the table, I3D, TANet, and ACTION-Net are isolated sign language word recognition methods based on traditional cameras, using video data as input. The I3D method uses a dense sampling method to obtain 64 frames from each video for training and recognition, and the TANet and ACTION-Net methods use a uniform sampling method to obtain 8 frames from each video for training and recognition; the isolated sign language word recognition method based on an event camera mentioned in the present invention uses an event stream as input and uses an event-driven sampling algorithm to obtain 8 key frames for training and recognition. All of the above methods use the same pre-trained model Kinetic400.

[0120] By comparing the experimental results in the table, it can be found that the isolated sign language word recognition method based on an event camera proposed in the present invention can use event stream data different from video as input for isolated sign language word recognition, successfully expanding the application of event cameras in the field of isolated sign language words; at the same time, compared with the isolated sign language word recognition method based on traditional cameras, the method proposed in the present invention has a higher recognition accuracy at the same or even fewer sampling frames, and the smaller the number of model convergence cycles, the higher the recognition efficiency. It can be proved that the present invention can effectively extract key information and high-speed motion information from isolated sign language words for recognition, overcoming the problem that the recognition effect of the existing technology is limited by video information redundancy and motion blur, and improving the recognition accuracy and efficiency.

[0121] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software or by using software plus the necessary general hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as ROM / RAM, CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for causing a computer device (such as a computer, server, smart phone or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0122] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for isolated sign language word recognition based on event camera, characterized in that: The following steps are involved: Step 1. Obtaining an isolated sign language word event stream: Use an event camera to re-shoot the isolated sign language word video. Before re-shooting, add padding and borders to the isolated sign language word video based on the resolution ratio of the isolated sign language word video, the display, and the event camera, and then stretch it to full screen. The relative position of the event camera and the display is fixed. When reshooting, the isolated sign language word video is played at the original frame rate, and the event camera outputs a cluster of events with positive and negative polarities in time sequence and stores them in array form to obtain the isolated sign language word event stream; Step 2: Generate low-speed and high-speed keyframe tensors: Slice the isolated sign language word event stream obtained in step 1 according to a fixed time window length to obtain multiple event stream slices, record the sequence number of the last event in each event stream slice in the isolated sign language word event stream to construct a sequence number array, count the number of events in each event stream slice to construct an event number array, sample the event stream slices using an event-driven sampling algorithm based on the event number array to obtain a set of keyframes, locate all events in each keyframe based on the sequence number array, and represent the set of keyframes in the form of a voxel grid using all events in each keyframe. During the representation, set a variable time block parameter. Setting the time block parameter indicates that the event distribution in a set of keyframes is counted at a low time resolution to obtain a low-speed keyframe tensor, and setting the time block parameter indicates that the event distribution in the same set of keyframes is counted at a high time resolution to obtain a high-speed keyframe tensor. Step 3. Use a convolutional neural network classification including a high- and low-speed motion information fusion module and a timing shift module to obtain the recognition result: first, pre-process the low-speed key frame tensor and the high-speed key frame tensor obtained in step 2 and input them into the high- and low-speed motion information fusion module to obtain a fused tensor; then input the fused tensor into the timing shift module to obtain a feature tensor, and finally send the feature tensor into the Softmax classifier to obtain the isolated sign language word recognition result.

2. The isolated sign language word recognition method based on event camera according to claim 1, characterized in that: In step 1: Step 11: Before reshooting, use the OPENCV image processing tool library to add padding and borders around each frame of the isolated sign language word video. The calculation formulas for the padding length h along the height direction and the padding length w along the width direction of the isolated sign language word video are as follows: Where H v and W v are the height and width of the isolated sign language word video resolution, H d and W d where ∫ is the height and width of the display resolution, and b is the border width. The isolated sign language video is then proportionally stretched to full screen until the height is the same as the display resolution, and offset by a certain length along the width to center the video on the display. The offset length is calculated as follows: Where H c and W c are the height and width of the event camera resolution respectively; the relative positions of the display and the event camera are fixed so that the boundary of the event camera imaging screen coincides with the border added to the isolated sign language word video; When reshooting, play the isolated word sign language video at the original frame rate, open the event camera, and store a cluster of events with positive and negative polarity output by the event camera in the form of an array to obtain the isolated sign language word event stream. Where (x k ,y k ,t k ,p k ) represents an event in the isolated sign language word event stream, k is the event sequence number, p k is the polarity of the event, t k is the timestamp, x k ,y k are the pixel horizontal and vertical coordinates respectively, and N is the total number of events.

3. The isolated sign language word recognition method based on event camera according to claim 1, characterized in that: In step 2: Step 21: Set a fixed time window length Length = 1 / P, where P is the frame rate of the isolated sign language word video. Slice the isolated sign language word event stream according to Length to obtain multiple event stream slices. e i Slice a certain event stream; record the last event of each event stream slice in the isolated sign language word event stream as l i , get the sequence number array The number of events recorded in each event stream slice is recorded as E i , get the event number array Where M is the total number of event stream slices, i is the index subscript of a certain event stream slice, i is an integer and 1≤i≤M; Step 22, acquiring key frames based on the event-driven sampling algorithm includes two parts: calculating the cumulative integral function of the event number array and sampling key frames according to the nearest neighbor principle; Count event count array The cumulative integral function D m , the formula is as follows: Where m is the index subscript of the event array element, which corresponds one-to-one to the event stream slice index subscript, and μ is a hyperparameter; According to the nearest neighbor principle, key frames are sampled, D m ∈(0,1], let the number of sampling frames be T, and divide the interval (0,1] into T segments equally Select a random value in each subinterval and find the cumulative integral function D of the event number array closest to the random value. m The event number array element index subscript corresponding to the function value is marked as the key frame index subscript f τ , the formula is as follows: Where, is a subinterval, τ is an integer and 1≤τ≤T, rand is a random value generation function, nearest is the nearest neighbor function, and samples are taken in each subinterval in turn. The event stream slice is indexed by the key frame index subscript to obtain a set of key frames recorded as in For a key frame; Step 23, according to the sequence number array and a set of keyframes A key frame All events in are represented as in, Keyframe The last event number in The last event sequence number of the key frame before this key frame, representing the key frame online in the form of a voxel grid First, the keyframe The timestamps of all events in k Linear shrinkage to [0, B-1], the formula is as follows: Where B is the time block parameter, represents the event stamp after linear shrinkage, Represents a keyframe The last event timestamp in Indicates the last event timestamp of the key frame before this key frame; then the key frame tensor V(b,y) in the form of a voxel grid is generated k ,x k ), the formula is as follows: Where b is an integer and b∈[0,B-1], k is the event number, p k is the event polarity, x k ,y k They are the horizontal and vertical coordinates of the pixel respectively, max represents the maximum value function, and the size of the generated key frame tensor is 2B×H×W. The first dimension 2B represents the channel dimension size, the coefficient 2 of 2B represents the positive and negative event polarity, B is the time block parameter, H and W represent the height and width of the spatial dimension respectively; for a set of key frames with a sampling frame number of T Setting the time block parameter B = 1 means that the event distribution in a set of key frames is counted at a low time resolution to obtain a low-speed key frame tensor V of size T×2×H×W. L Setting the time block parameter B = 4 means that the event distribution in a set of key frames is counted at a high time resolution to obtain a high-speed key frame tensor V of size T×8×H×W H .

4. The isolated sign language word recognition method based on event camera according to claim 1, characterized in that: In step 3: Step 31: The low-speed and high-speed key frame tensors V obtained in step 2 are L 、V H The two-dimensional convolution layer and the maximum pooling layer are respectively used to achieve dimensionality increase in the channel dimension and dimensionality reduction in the spatial dimension, and the low-speed key frame tensor U with a size of T×64×H / 4×W / 4 is obtained. L and the preprocessed high-speed keyframe tensor U with size T×256×H / 4×W / 4 H : U L ,U H =Maxpool(Conv(V L )),Maxpool(Conv(V H )) (8) In the formula, Conv represents the two-dimensional convolution layer, Maxpool represents the maximum pooling layer; Will U L 、U H The high-speed and low-speed motion information fusion module is firstly used to fusion the pre-processed high-speed key frame tensor U H Perform a global average pooling operation in the spatial domain to make the size of the spatial dimension 1, achieve spatial compression, and obtain the compressed high-speed key frame tensor Then It is sent to the fully connected layer, and the channel dimension is reduced by the ReLU activation layer, and then sent to the fully connected layer and the Sigmoid activation layer to finally make Channel dimension and U L Consistent, get the high-speed key frame tensor after excitation Finally, the high-speed key frame tensor after excitation and the preprocessed low-speed key frame tensor U L Perform matrix multiplication and then add it to the preprocessed low-speed key frame tensor U L Add together to get the fused tensor U of size T×64×H / 4×W / 4 F : Where GAP represents global average pooling, W1 and W2 represent fully connected layers, δ and σ represent ReLU activation function and Sigmoid activation function respectively, and ⊙ represents matrix element multiplication. Step 32: Fuse the fused tensor U L Input the timing shift module to obtain the feature tensor. The timing shift module consists of two parts: timing shift operation and backbone network. The standard Resnet-50-2D is selected as the backbone network. The Resnet-50-2D backbone network contains four Bottleneck feature extraction subnetworks. Each Bottleneck feature extraction subnetwork extracts features from the fused tensor in different spatial dimensions. The temporal shift operation shifts the fused tensor input into each Bottleneck feature extraction subnetwork in the time dimension, that is, the temporal shift operation is embedded in the heads of the four Bottleneck feature extraction subnetworks. Assuming that the number of channels contained in the fused tensor input into a Bottleneck feature extraction subnetwork is in_channels and the shift depth is d, the number of channels c shifted in one direction is: c=in_channels / d (12) The fused tensor is split into three parts in the channel dimension: the left end, the middle end, and the right end. The left end contains the first c channels, the middle end contains the cth to 2cth channels, and the remaining channels are the right end. The left end is shifted one unit to the left in the time dimension, and the middle end is shifted one unit to the right in the time dimension. The vacant dimension is filled with zeros to establish a bidirectional temporal relationship. The right end remains unchanged to maintain the stability of its own spatial structure. The formula is as follows: Where, Represents the left, middle, and right ends of the split fusion tensor. Lshift and Rshift represent left and right shift operations respectively. Concat represents reconnecting the split fusion tensor to obtain the fusion tensor to be input into a Bottleneck feature extraction subnetwork after time sequence shifting. Fused Tensor U F After four Bottleneck feature extraction sub-networks embedded with temporal shift operations, the output sizes become T×256×H / 4×W / 4, T×512×H / 8×W / 8, T×512×H / 16×W / 16, and T×2048×H / 32×W / 32 respectively. The output of the last Bottleneck feature extraction sub-network is the feature tensor output by the temporal shift module, which is recorded as Finally, the feature tensor The feature tensor is fed into the Softmax classifier, which consists of an average pooling layer, a linear fully connected layer, and a Softmax layer. After the average pooling layer and the linear fully connected layer, the size of the feature tensor becomes T × classes. The class dimension size corresponds to the total number of isolated sign language word categories, and the value is the score of each category. Then, the T scores of each category are averaged to return the feature tensor size to classes. After the final Softmax layer sorts the scores, the index corresponding to the maximum score is output as the isolated sign language word recognition result: In the formula, Avgpool is the average pooling layer, Linear is the linear fully connected layer, Average is the average operation, and Softmax is the Softmax layer.

5. An electronic device, characterized in that: including processor and memory; Memory for storing computer programs; A processor, configured to execute a computer program stored in a memory, and to implement the method according to any one of claims 1 to 4 during execution.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images