Image spatial-temporal feature extraction method, gait recognition method, equipment and medium

Abstract

The invention discloses an image spatial-temporal feature extraction method, a gait recognition method, equipment and a medium, and the method comprises the steps: obtaining a to-be-processed image frame sequence, and converting each to-be-processed image frame in the to-be-processed image frame sequence into an embedded vector through an embedded coding layer; inputting the embedded vector into a spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to the to-be-processed image frame sequence; inputting the spatial feature vector into a time sequence feature extraction layer for time sequence feature extraction to obtain a time sequence feature vector corresponding to the to-be-processed image frame sequence; and fusing the spatial feature vector and the time sequence feature vector by using a spatial-temporal feature fusion layer to obtain a spatial-temporal feature vector corresponding to the to-be-processed image frame sequence. According to the method, local detail feature extraction and global time sequence feature extraction are optimized, and the representation capability and accuracy of the spatial-temporal feature vector are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image spatio-temporal feature extraction method, a gait recognition method, equipment and a medium. BACKGROUND

[0002] Image spatio-temporal feature is an important concept in the field of image recognition, aiming to capture and analyze the rules of object changes over time and space in images, so as to better understand and predict the behavior of objects in dynamic scenes. Image spatio-temporal feature has wide application prospects in the fields of video analysis, autonomous driving, human-computer interaction, etc.

[0003] In the prior art, convolutional neural networks (CNN) or Transformer networks are generally used for spatio-temporal feature extraction. They can model the spatio-temporal features of short sequence images well, but their modeling ability for long sequence images is insufficient, which affects the accuracy of image recognition algorithms. SUMMARY

[0004] To solve the above technical problems, the present application at least provides an image spatio-temporal feature extraction method, a gait recognition method, equipment and a medium.

[0005] The first aspect of the present application provides an image spatio-temporal feature extraction method applied to a spatio-temporal feature extraction network, the spatio-temporal feature extraction network containing an embedding encoding layer, a spatial feature extraction layer, a time sequence feature extraction layer and a spatio-temporal feature fusion layer, the spatial feature extraction layer being composed of a plurality of cascaded Transformer Blocks, the time sequence feature extraction layer being composed of a plurality of cascaded Mamba Blocks, the method comprising: obtaining a sequence of image frames to be processed, and converting each image frame to be processed in the sequence of image frames to be processed into an embedding vector by using the embedding encoding layer; wherein the sequence of image frames to be processed contains a plurality of image frames to be processed sorted in time sequence; inputting the embedding vector into the spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to the sequence of image frames to be processed; inputting the spatial feature vector into the time sequence feature extraction layer for time sequence feature extraction to obtain a time sequence feature vector corresponding to the sequence of image frames to be processed; and fusing the spatial feature vector and the time sequence feature vector by using the spatio-temporal feature fusion layer to obtain a spatio-temporal feature vector corresponding to the sequence of image frames to be processed.

[0006] In an embodiment, converting each image frame to be processed in the sequence of image frames to be processed into an embedding vector by using the embedding encoding layer comprises: performing image block segmentation on each image frame to be processed in the sequence of image frames to be processed to obtain a plurality of image blocks; and respectively converting each image block into an embedding vector to generate an embedding vector of each image block.

[0007] In an embodiment, the embedding encoding layer comprises a three-dimensional linear projection layer and a multi-layer perception machine; converting each to-be-processed image frame in the sequence of to-be-processed image frames into an embedding vector by using the embedding encoding layer comprises: performing image block segmentation on each to-be-processed image frame respectively by using the three-dimensional linear projection layer, and performing flattening and mapping to a channel dimension on each image block to obtain an initial vector corresponding to each image block respectively; and mapping the initial vector corresponding to each image block respectively by using the multi-layer perception machine to obtain an embedding vector corresponding to each image block respectively.

[0008] In an embodiment, the method further comprises: extracting image features of multiple modalities of each to-be-processed image frame to obtain a sequence of feature maps of multiple modalities; performing image block segmentation on each to-be-processed image frame respectively by using the three-dimensional linear projection layer, and performing flattening and mapping to a channel dimension on each image block to obtain an initial vector of multiple modalities corresponding to each image block respectively, comprising: performing image block segmentation on the sequence of feature maps of multiple modalities respectively by using the three-dimensional linear projection layer, and performing flattening and mapping to a channel dimension on each image block to obtain an initial vector of multiple modalities corresponding to each image block respectively; and mapping the initial vector corresponding to each image block respectively by using the multi-layer perception machine to obtain an embedding vector corresponding to each image block respectively, comprising: splicing the initial vector of multiple modalities corresponding to each image block respectively to obtain a multi-modal spliced vector corresponding to each image block respectively; and mapping the multi-modal spliced vector corresponding to each image block respectively by using the multi-layer perception machine to obtain an embedding vector corresponding to each image block respectively.

[0009] In an embodiment, the embedding vector is input into a spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to each image block, comprising: inputting the embedding vector of each image block into the spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to each image block; inputting the spatial feature vector into a temporal feature extraction layer for temporal feature extraction to obtain a temporal feature vector corresponding to the sequence of to-be-processed image frames, comprising: splicing the spatial feature vectors of the image blocks belonging to the same to-be-processed image frame to obtain an embedding vector of each to-be-processed image frame, and sorting the embedding vectors of each to-be-processed image frame based on the time sequence of each to-be-processed image frame to obtain a vector sorting result; and inputting the vector sorting result into the temporal feature extraction layer for temporal feature extraction to obtain a temporal feature vector corresponding to the sequence of to-be-processed image frames.

[0010] The second aspect of the present application provides a gait recognition method, the method comprising: acquiring a sequence of gait image frames obtained by image acquisition on a target object; performing spatio-temporal feature extraction on the sequence of gait image frames based on a spatio-temporal feature extraction network to obtain a spatio-temporal feature vector corresponding to the target object in the sequence of gait image frames; and performing gait classification on the target object based on the spatio-temporal feature vector to obtain a gait recognition result corresponding to the target object.

[0011] In an embodiment, the method further comprises: extracting image features of the target object in the gait image frame sequence in multiple modalities to obtain a feature map sequence corresponding to the target object in multiple modalities; and performing spatio-temporal feature extraction on the gait image frame sequence based on a spatio-temporal feature extraction network to obtain a spatio-temporal feature vector corresponding to the target object in the gait image frame sequence, including: inputting the feature map sequence corresponding to the target object in multiple modalities into the spatio-temporal feature extraction network for spatio-temporal feature extraction to obtain the spatio-temporal feature vector corresponding to the target object in the gait image frame sequence.

[0012] In an embodiment, the extracting image features of the target object in the gait image frame sequence in multiple modalities to obtain a feature map sequence corresponding to the target object in multiple modalities comprises: extracting a gait feature map of the target object, and / or a movable part analysis feature map of the target object, and / or a key point feature map of the target object in the gait image frame sequence to obtain a gait feature map sequence, and / or a movable part analysis feature map sequence, and / or a key point feature map sequence of the target object in the gait image frame sequence.

[0013] The third aspect of the present application provides an image spatio-temporal feature extraction device, the device comprising: an image embedding module configured to obtain a sequence of image frames to be processed, and convert each image frame to be processed in the sequence of image frames to be processed into an embedding vector using an embedding coding layer; wherein the sequence of image frames to be processed comprises a plurality of image frames to be processed sorted in time order; a spatial feature extraction module configured to input the embedding vector into a spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to the sequence of image frames to be processed; a time sequence feature extraction module configured to input the spatial feature vector into a time sequence feature extraction layer for time sequence feature extraction to obtain a time sequence feature vector corresponding to the sequence of image frames to be processed; and a feature fusion module configured to fuse the spatial feature vector and the time sequence feature vector using a spatio-temporal feature fusion layer to obtain a spatio-temporal feature vector corresponding to the sequence of image frames to be processed.

[0014] The fourth aspect of the present application provides a gait recognition device, the device comprising: an image acquisition module configured to obtain a sequence of gait image frames obtained by image acquisition of a target object; a feature extraction module configured to perform spatio-temporal feature extraction on the sequence of gait image frames based on a spatio-temporal feature extraction network to obtain a spatio-temporal feature vector corresponding to the target object in the sequence of gait image frames; and a gait recognition module configured to perform gait classification on the target object based on the spatio-temporal feature vector to obtain a gait recognition result corresponding to the target object.

[0015] The fifth aspect of the present application provides an electronic device comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the above-mentioned image spatio-temporal feature extraction method or gait recognition method.

[0016] The sixth aspect of the present application provides a computer readable storage medium, which stores program instructions, and the program instructions are executed by a processor to implement the image spatio-temporal feature extraction method or the gait recognition method.

[0017] The above scheme, by acquiring a to-be-processed image frame sequence, converting each to-be-processed image frame in the to-be-processed image frame sequence into an embedding vector by using an embedding coding layer; inputting the embedding vector into a spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to the to-be-processed image frame sequence; inputting the spatial feature vector into a time sequence feature extraction layer for time sequence feature extraction to obtain a time sequence feature vector corresponding to the to-be-processed image frame sequence; and fusing the spatial feature vector and the time sequence feature vector by using a spatio-temporal feature fusion layer to obtain a spatio-temporal feature vector corresponding to the to-be-processed image frame sequence, first using the characteristic that the Transformer Block is good at local feature extraction to realize efficient representation of the spatial information of a single image frame, and then using the state space mechanism of the Mamba Block to globally model the time sequence dependence of the to-be-processed image frame sequence, optimizing local detailed feature extraction and global time sequence feature extraction, and improving the representation ability and accuracy of the spatio-temporal feature vector.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0020] Figure 1 is a schematic diagram of a scheme implementation environment shown by an exemplary embodiment of the present application;

[0021] Figure 2 is a network architecture diagram of a spatio-temporal feature extraction network shown by an exemplary embodiment of the present application;

[0022] Figure 3 is a flowchart of an image spatio-temporal feature extraction method shown by an exemplary embodiment of the present application;

[0023] Figure 4 is a flowchart of a gait recognition method shown by an exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a gait feature map sequence shown by an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a movable part analysis feature map sequence shown by an exemplary embodiment of the present application;

[0026] Figure 7 FIG. 1 is a schematic diagram of a keypoint feature map sequence according to an example embodiment of the present application;

[0027] Figure 8 FIG. 2 is a schematic diagram of a spatio-temporal feature extraction network according to another example embodiment of the present application;

[0028] Figure 9 FIG. 3 is a schematic diagram of a Transformer Block according to an example embodiment of the present application;

[0029] Figure 10 FIG. 4 is a schematic diagram of a Mamba Block according to an example embodiment of the present application;

[0030] Figure 11 FIG. 5 is a block diagram of an image spatio-temporal feature extraction apparatus according to an example embodiment of the present application;

[0031] Figure 12 FIG. 6 is a block diagram of a gait recognition apparatus according to an example embodiment of the present application;

[0032] Figure 13 FIG. 7 is a structural schematic diagram of an electronic device according to an example embodiment of the present application;

[0033] Figure 14 FIG. 8 is a structural schematic diagram of a computer readable storage medium according to an example embodiment of the present application. DETAILED DESCRIPTION

[0034] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0035] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, rather than for the sake of limitation.

[0036] The term "and / or" herein is merely a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents that the front and rear associated objects have an "or" relationship. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0037] The current mainstream spatio-temporal feature extraction method is mainly based on convolutional neural network (CNN) and Transformer architecture, but the applicant finds that multiple technical bottlenecks are faced in practical application.

[0038] Firstly, the CNN model has insufficient modeling capability for dynamic changes of long sequence images, and when fusing multi-modal image data, a simple cascading strategy is usually adopted, which is difficult to effectively balance the temporal dependence and cross-modal feature expression.

[0039] Secondly, although the self-attention mechanism of the Transformer can capture global spatio-temporal features, in the analysis scene of long sequence images, the secondary computational complexity and memory overhead caused by the self-attention mechanism of the Transformer lead to unstable training and difficulty in effectively modeling global spatio-temporal dependence, and the actual attention distribution is prone to deviate to local areas.

[0040] In order to solve the above problems, the present application provides at least an image spatio-temporal feature extraction method, a gait recognition method, a device and a medium.

[0041] The image spatio-temporal feature extraction method and the gait recognition method provided by the embodiments of the present application will be described below.

[0042] Please refer to Figure 1 , Figure 1 is a schematic diagram of a scheme implementation environment shown by an exemplary embodiment of the present application. The scheme implementation environment can include a terminal 110 and a server 120, which are connected in communication with each other.

[0043] The number of terminals 110 can be one or more. The terminal 110 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0044] The server 120 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0045] In one example, a spatio-temporal feature extraction network is deployed in the server 120, the server 120 obtains a to-be-processed image frame sequence from the terminal 110, and performs image spatio-temporal feature extraction processing on the to-be-processed image frame sequence based on the spatio-temporal feature extraction network to obtain a spatio-temporal feature vector. Of course, the server 120 can store the spatio-temporal feature vector locally, return it to the terminal 110, or transmit it to other terminals.

[0046] Specifically, refer to Figure 2 , Figure 2 is a network architecture diagram of a spatio-temporal feature extraction network according to an example embodiment of the present application, as Figure 2 shown, the spatio-temporal feature extraction network contains an embedding encoding layer, a spatial feature extraction layer, a time sequence feature extraction layer, and a spatio-temporal feature fusion layer, the spatial feature extraction layer is composed of multiple cascaded Transformer Blocks, the time sequence feature extraction layer is composed of multiple cascaded Mamba Blocks, the embedding encoding layer is used for embedding vector conversion, the spatial feature extraction layer is mainly used for extracting a spatial feature vector, the time sequence feature extraction layer is mainly used for extracting a time sequence feature vector, and the spatio-temporal feature fusion layer is used for fusing the spatial feature vector and the time sequence feature vector to obtain a final spatio-temporal feature vector.

[0047] In one example, the server 120 can also perform gait recognition on the gait image frame sequence obtained from the terminal 110 to obtain a gait recognition result, and of course, the server 120 can store the gait recognition result locally, return it to the terminal 110, or transmit it to other terminals.

[0048] In one example, the terminal 110 is installed and runs a client of a target application, such as an application that provides image spatio-temporal feature extraction and / or gait recognition functions. The server 120 can be a background server of the target application, used to provide background services for the client of the target application.

[0049] The image spatio-temporal feature extraction method or gait recognition method provided by the embodiments of the present application can be executed by the terminal 110, such as a client of a target application installed and run in the terminal 110, or by the server 120, or by the terminal 110 and the server 120 in cooperation, that is, part of the steps of the method are executed by the terminal 110 and the other part of the steps are executed by the server 120.

[0050] Please refer to Figure 3 , Figure 3 is a flowchart of an image spatio-temporal feature extraction method according to an example embodiment of the present application. The image spatio-temporal feature extraction method can be applied to Figure 1 the implementation environment shown, and is specifically executed by the server in the implementation environment. It should be understood that the method can also be applied to other example implementation environments, and is specifically executed by devices in other implementation environments, and the embodiments of the present application do not limit the implementation environment to which the method is applied.

[0051] As Figure 3 shown, the image spatio-temporal feature extraction method at least includes steps S310 to S340, which are described in detail as follows:

[0052] Step S310: Obtain a sequence of to-be-processed image frames, and convert each to-be-processed image frame in the sequence of to-be-processed image frames into an embedding vector by using an embedding encoding layer.

[0053] The sequence of to-be-processed image frames contains a plurality of to-be-processed image frames sorted in time sequence. The to-be-processed image frames can be visible light images (such as RGB images) or infrared images, and the type of the to-be-processed image frames is not limited in the present application.

[0054] Each to-be-processed image frame in the sequence of to-be-processed image frames is input into the embedding encoding layer to convert the image information of the to-be-processed image frame into a digital vector that can be understood by a computer, thereby obtaining an embedding vector corresponding to the to-be-processed image frame.

[0055] Step S320: Input the embedding vector into a spatial feature extraction layer for spatial feature extraction, thereby obtaining a spatial feature vector corresponding to each image block.

[0056] The Transformer Block in the spatial feature extraction layer is used for extracting spatial features of an image. The spatial features of the image refer to attributes related to spatial distribution of the image, such as positions, shapes, spatial relationships and local structures of pixels in the image.

[0057] The Transformer Block directly models global dependency relationships between image contents at different positions of the image based on a self-attention mechanism (Self-Attention), and realizes efficient representation of spatial features of a single image by using the characteristics of local feature extraction.

[0058] The embedding vector is input into the spatial feature extraction layer for spatial feature extraction, thereby obtaining a spatial feature vector corresponding to each to-be-processed image frame output by the spatial feature extraction layer.

[0059] Step S330: Input the spatial feature vector into a temporal feature extraction layer for temporal feature extraction, thereby obtaining a temporal feature vector corresponding to the sequence of to-be-processed image frames.

[0060] The Mamba Block in the temporal feature extraction layer is used for extracting temporal features of an image. The temporal features of the image refer to information of the image changing with time, such as position information, posture information and action information.

[0061] The Mamba Block globally models long-sequence temporal dependencies based on a state space model (SSM), thereby realizing efficient representation of temporal features of long-sequence to-be-processed image frames.

[0062] The spatial feature vector is input into a time sequence feature extraction layer for time sequence feature extraction, to obtain a time sequence feature vector corresponding to the image frame sequence to be processed output by the time sequence feature extraction layer.

[0063] Step S340: The spatial feature vector and the time sequence feature vector are fused by using a space-time feature fusion layer to obtain a space-time feature vector corresponding to the image frame sequence to be processed.

[0064] The fused spatial feature vector and time sequence feature vector are fused, and the fused feature vector is taken as a space-time feature vector corresponding to the image frame sequence to be processed.

[0065] The present application first uses the characteristic of the Transformer Block being good at local feature extraction to realize efficient representation of the spatial information of a single image frame, and then uses the state space mechanism of the Mamba Block to globally model the time sequence dependence of the image frame sequence to be processed, thereby optimizing local detailed feature extraction and global time sequence feature extraction, overcoming the quadratic complexity bottleneck and the defect of paying too much attention to local details of the traditional Transformer in a long sequence scenario, and breaking through the limitation of the depth of time sequence modeling of the traditional model.

[0066] Next, some embodiments of the present application will be described in detail.

[0067] In some embodiments, the conversion of each image frame to be processed in the image frame sequence to be processed into an embedding vector in step S310 includes the following steps S311 to S312.

[0068] Step S311: Each image frame to be processed in the image frame sequence to be processed is subjected to image block segmentation to obtain a plurality of image blocks.

[0069] Each image frame to be processed in the image frame sequence to be processed is subjected to image block segmentation to obtain a plurality of image blocks.

[0070] For example, each image frame to be processed is uniformly segmented in a 1x14x14 size sliding window to obtain a plurality of image blocks.

[0071] Step S312: Each image block is respectively converted into an embedding vector to generate an embedding vector of each image block.

[0072] Each image block is respectively converted into an embedding vector.

[0073] Exemplarily, the embedding coding layer comprises a three-dimensional linear projection layer (Three-Dimensional Linear Projection of Flattened Patches) and a multilayer perceptron (MLP), the three-dimensional linear projection layer is used to respectively perform image block segmentation on each image frame to be processed, and each image block is flattened and mapped to a channel dimension to obtain an initial vector corresponding to each image block, and the multilayer perceptron is used to map the initial vector corresponding to each image block to obtain an embedding vector corresponding to each image block.

[0074] Optionally, image features of multiple modalities of each image frame to be processed can also be extracted to obtain a feature map sequence of multiple modalities, the three-dimensional linear projection layer is used to respectively perform image block segmentation on the feature map sequence of multiple modalities, and each image block is flattened and mapped to a channel dimension to obtain an initial vector of multiple modalities corresponding to each image block, then, the initial vector of multiple modalities corresponding to each image block is spliced to obtain a multi-modal splicing vector corresponding to each image block, and the multilayer perceptron is used to map the multi-modal splicing vector corresponding to each image block to obtain an embedding vector corresponding to each image block.

[0075] The image features of multiple modalities refer to image information extracted in different forms, such as contour information, texture information, key point information, etc. of a specified object in an image frame, and the type of image features to be extracted can be flexibly selected according to a specific application scenario to obtain image features of multiple modalities.

[0076] After obtaining the embedding vector of each image block based on the above embodiment, the embedding vector of each image block is input into a spatial feature extraction layer for spatial feature extraction to obtain a spatial feature vector corresponding to each image block.

[0077] Then, the spatial feature vector corresponding to each image block is rearranged according to a space-time relationship, specifically, the spatial feature vectors of image blocks belonging to the same image frame to be processed are spliced to obtain an embedding vector of each image frame to be processed, and the embedding vectors of each image frame to be processed are sorted based on a time sequence of each image frame to be processed to obtain a vector sorting result.

[0078] The vector sorting result is then input into a time sequence feature extraction layer for time sequence feature extraction to obtain a time sequence feature vector corresponding to the image frame sequence.

[0079] Finally, the spatial feature vector and the time sequence feature vector are fused to obtain a space-time feature vector corresponding to the image frame sequence.

[0080] The obtained spatio-temporal feature vector can be used for gait recognition, action recognition, motion trajectory recognition, event recognition, etc., which are not limited in the present application.

[0081] Next, the present application scheme is described in detail taking gait recognition as an example.

[0082] Please refer to Figure 4 , Figure 4 is a flowchart of a gait recognition method shown in an exemplary embodiment of the present application. The gait recognition method can be applied to Figure 1 the implementation environment shown in the figure and specifically executed by the server in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and specifically executed by devices in other implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.

[0083] As shown in Figure 4 , the gait recognition method at least includes steps S410 to S430, which are described in detail as follows:

[0084] Step S410: obtaining a gait image frame sequence obtained by image acquisition on a target object.

[0085] Among them, the target object refers to the object that needs to be gait recognized.

[0086] For example, the gait image frame is an RGB image, and the gait image frame sequence is represented as X ∈ R B×T×3×H×W , where B represents batch, specifically used to represent the number of images input at a time; T represents the length of the gait image frame sequence, i.e. the number of gait image frames contained in the gait image frame sequence; H represents the height of the gait image frame, and W represents the width of the gait image frame; 3 is the red (Red), green (Green), and blue (Blue) three color channels corresponding to the RGB image.

[0087] Step S420: based on the spatio-temporal feature extraction network, performing spatio-temporal feature extraction on the gait image frame sequence to obtain a spatio-temporal feature vector corresponding to the target object in the gait image frame sequence.

[0088] The spatio-temporal feature extraction network in the above embodiment is used to perform spatio-temporal feature extraction on the gait image frame sequence to obtain a spatio-temporal feature vector corresponding to the target object in the gait image frame sequence.

[0089] In some embodiments, the image features of multiple modalities of the target object in the gait image frame sequence are extracted to obtain a feature map sequence of multiple modalities corresponding to the target object, and the feature map sequence of multiple modalities corresponding to the target object is input into the spatio-temporal feature extraction network for spatio-temporal feature extraction to obtain a spatio-temporal feature vector corresponding to the target object in the gait image frame sequence.

[0090] For example, the gait feature map of the target object in the gait image frame sequence, and / or the movable part analysis feature map of the target object, and / or the key point feature map of the target object are extracted, to obtain the corresponding gait feature map sequence, and / or the movable part analysis feature map sequence, and / or the key point feature map sequence of the target object in the gait image frame sequence.

[0091] For example, the gait image frame sequence is represented as X∈R B×T×3×H×W For example, the gait image frame sequence is represented as X∈R Figure 5 , Figure 5 is a schematic diagram of the gait feature map sequence shown in an example embodiment of the present application, as shown in Figure 5 The contour information and motion information (such as stride width) of the target object in each gait image frame are extracted to obtain the gait feature map of the target object, and the gait feature map of each gait image frame is combined to obtain the gait feature sequence S∈R B×T×1×H×W .

[0092] For example, the gait image frame sequence is represented as X∈R Figure 6 , Figure 6 is a schematic diagram of the movable part analysis feature map sequence shown in an example embodiment of the present application, as shown in Figure 6 The motion information (such as swing amplitude and tilt angle) of the movable part of the target object in each gait image frame is extracted to obtain the movable part analysis feature map of the target object, and the movable part analysis feature map of each gait image frame is combined to obtain the movable part analysis feature map sequence P∈R B×T×3×H×W .

[0093] For example, the gait image frame sequence is represented as X∈R Figure 7 , Figure 7 is a schematic diagram of the key point feature map sequence shown in an example embodiment of the present application, as shown in Figure 7 The dynamic change information (such as angle change of key points between multiple image frames and gait cycle phase) of the key points of the target object in each gait image frame is extracted to obtain the key point feature map of the target object, and the key point feature map of each gait image frame is combined to obtain the key point feature map sequence K∈R B×T×3×H×W .

[0094] The embedding encoding layer of the spatio-temporal feature extraction network is used to convert the gait feature sequence S∈R B×T×1×H×W , the movable part analysis feature map sequence P∈R B×T×3×H×W , and the key point feature map sequence K∈R B×T×3×H×W into embedding vectors.

[0095] For example, the gait image frame sequence is represented as X∈R Figure 8 , Figure 8 is a schematic diagram of the spatio-temporal feature extraction network shown in another example embodiment of the present application, as shown in Figure 8As shown, the embedding coding layer consists of three 3D linear projection layers and one multilayer perceptron, which encodes the gait feature sequence S∈R. B×T×1×H×W Analytical feature map sequence P∈R of movable parts B×T×3×H×W Key point feature map sequence K∈R B ×T×3×H×W The inputs are fed into the corresponding 3D linear projection layers. The 3D linear projection layers use a 1×14×14 sliding window to uniformly segment the input feature maps. Then, the segmented image patches are flattened and mapped to dimension C, yielding the initial vectors S for different modalities corresponding to the image patches. p ∈R B×N×C P p ∈R B×N×C and K p ∈R B×N×C Where N refers to the total number of image patches obtained from segmentation. C represents the number of channels.

[0096] Then, S p P p and K p A multilayer perceptron (MLP) is used for mapping to obtain the multimodal embedding vector X corresponding to each image patch. mp ∈R B×N×C =MLP([X p ,P p H p ]), where [] represents the splicing operation.

[0097] Furthermore, the embedding vector of each image patch is input into the spatial feature extraction layer to extract spatial features, thereby obtaining the spatial feature vector corresponding to each image patch.

[0098] Specifically, the multimodal embedding vector X is obtained based on the above embodiments. mp Afterwards, in X mp A global gait feature vector (Classification Token, CLS Token), denoted as X, is concatenated into the head. cls ∈R B×1×C And add a position embedding, denoted as E. pos ∈R B×(N+1)×C This yields a new embedding vector Z, as detailed in Formula 1 below:

[0099]

[0100] Where N is the total number of image blocks obtained from the segmentation.

[0101] Among them, the global gait feature vector X cls and position code Epos is a learnable vector in the training process of the spatio-temporal feature extraction network.

[0102] The new embedding vector Z is input into the Transformer Block of the spatial feature extraction layer.

[0103] For example, Figure 9 is a schematic diagram of the Transformer Block shown in an exemplary embodiment of the present application, as Figure 9 shown, the Transformer Block contains a multi-head self-attention layer (Multi-Head Self Attention), a normalization layer (Layer Normalization), and a feed-forward neural network layer (Feed-Forward Neural Network).

[0104] After each Transformer Block, the corresponding operation can be as follows Formula 2 and public

[0105] Formula 3:

[0106]

[0107] Wherein, MHSA represents the relevant calculation of the multi-head self-attention layer, LN represents the relevant calculation of the layer normalization layer, and FFN represents the relevant calculation of the feed-forward neural network; Z i-1 represents the output of the i-1th Transformer Block, represents the output of the MHSA in the i th Transformer Block, and Z i represents the output of the i th Transformer Block.

[0108] After the operation of the Transformer Block in the spatial feature extraction layer is performed k times, the output Z k of the k th Transformer Block is obtained, which is sent into the Mamba Block at the same time as the m th Transformer Block is sent in. The Transformer Block and the Mamba Block perform double-branch parallel calculation to improve the calculation efficiency. Wherein, k can be pre-set according to experience, or can be flexibly determined according to the image quality of the gait image frame sequence, such as the higher the image quality, the earlier the output of the Transformer Block is selected as the input of the Mamba Block, and the lower the image quality, the later the output of the Transformer Block is selected as the input of the Mamba Block, to balance the calculation efficiency and the calculation accuracy.

[0109] Of course, the output of the last Transformer Block can also be directly selected as the input of the Mamba Block to maximize the gait recognition accuracy.

[0110] Z k First, the global gait feature vector X cls added by the Transformer Block is removed, and then rearranged according to the space-time relationship (for the specific implementation of the rearrangement, please refer to the above step S212) to obtain Next, a spatial pooling operation is performed to compress into X t ∈R B×T×1×C , which is used to represent the time sequence feature of the complete gait sequence of a single target object.

[0111] A time sequence feature vector (Time ClS Token) is added to the time sequence feature X t of the above gait sequence, denoted as Then, a time position encoding (Time Position Embed) is added, denoted as to obtain a time embedding vector Z t , for details, please refer to the following formula 4:

[0112]

[0113] Where T is the total number of gait image frames in the gait image frame sequence.

[0114] Where the time sequence feature vector and the time position encoding are learnable vectors in the training process of the space-time feature extraction network.

[0115] As shown in Figure 8 , the time sequence feature extraction layer is composed of j Mamba Blocks, and the time embedding vector Z t is input into the j Mamba Blocks.

[0116] For example, Figure 10 is a schematic diagram of the Mamba Block shown in an exemplary embodiment of the present application, as Figure 10As shown, the Mamba Block contains a normalization layer, a projection layer, a convolutional layer, a sigmoid-weighted linear unit activation function layer, and a state space model (SSM) layer.

[0117] For each Mamba Block traversed, the corresponding operation can be represented by the following formula 5:

[0118]

[0119] Where SSM represents the computational operation of the state-space model, SiLU is the computational operation of the activation function, Conv is the convolution operation of the one-dimensional convolutional network layer, Proj is the computational operation of the projection layer, and LN is the normalization operation. This represents the output of the (i-1)th Mamba Block. This represents the output of the i-th Mamba Block.

[0120] While the Mamba Block is extracting features, the next m Transformer Blocks will be executed synchronously according to Equations 2 and 3, ultimately yielding the outputs of the last Transformer Blocks and the Mamba Blocks, denoted as spatial feature vectors ZP∈R. B×(N+1)×C and time series eigenvectors ZT∈R B×(T+1)×C .

[0121] Finally, a spatiotemporal feature fusion layer is used to fuse spatial feature vectors and temporal feature vectors.

[0122] Specifically, such as Figure 8 As shown, the spatiotemporal feature fusion layer contains a multilayer perceptron (MLP). The outputs of the two branches are fused using the MLP to obtain the final spatiotemporal feature vector F∈R. B×D Where D represents the number of channels, the corresponding operation can be shown in Formula 6 below:

[0123] F = MLP([ZP, ZT]) (Formula 6)

[0124] The above embodiments utilize different Tokenization methods to unify the gait feature map sequence, the movable part parsing feature map sequence, and the key point feature map sequence into the same feature representation space, provide rich semantic and motion information, and then utilize the multi-modal fusion capability of the Transformer to improve the gait recognition performance of the model. The model structure is simplified, the model parameter amount is reduced, the alignment difficulty of different modalities is reduced, and the subsequent gait recognition accuracy and generalization are improved.

[0125] Step S430: performing gait classification on the target object based on the spatiotemporal feature vector to obtain a gait recognition result corresponding to the target object.

[0126] As shown in FIG. 13, after obtaining the spatiotemporal feature vector, the spatiotemporal feature vector is input into a fully connected head (FC Head) for gait classification to obtain a gait recognition result corresponding to the target object. Figure 8

[0127] In some embodiments, during the model training process of the spatiotemporal feature extraction network, after obtaining the spatiotemporal feature vector, a model training loss calculation process is further included, and the model training loss can be calculated based on a triplet loss, a cross-entropy loss function, or the like.

[0128] For example, the triplet loss L tri is calculated based on the spatiotemporal feature vector F tri . The formula for calculating the triplet loss L tri is shown in the following formula 7:

[0129] L i = max(S(F i+ ,F i )-S(F i- ,F i + α, 0) (formula 7)

[0130] where S(·) represents a feature vector distance calculation function, F i+ represents the spatiotemporal feature vector corresponding to the i-th sample, F i- and F i represent the feature vectors of the samples belonging to the same class and different classes, respectively, and α is a preset boundary value to avoid interference of abnormal values on training.

[0131] For another example, the cross-entropy loss function is calculated based on the spatiotemporal feature vector F. To calculate the cross-entropy loss function, the spatiotemporal feature vector F is first input into the fully connected head (FC Head) to obtain a corresponding classification probability F id ∈ R B×R×D ​wherein R represents the number of categories, and the cross-entropy loss function L is calculated id The formula is expressed as the following formula 8:

[0132]

[0133] wherein q r represents the true classification label, and p r represents the classification prediction probability of the model.

[0134] By combining Transformer and Mamba, the perception ability of subtle changes in the gait recognition cycle of the target object is improved, and then the accuracy and generalization of the gait recognition task are enhanced. Among them, Transformer is based on the self-attention mechanism, directly models the global dependency relationship between all image blocks in the image, can capture the cooperative motion mode of different parts in the gait sequence at the same time, and has efficient multi-modal information fusion capability. Mamba is based on a state space model, and its computational complexity grows linearly with the sequence length, which is suitable for processing high-frame-rate long-sequence data such as gait videos. Through the selective state space mechanism, Mamba can dynamically adjust the parameters to accurately capture the timing change characteristics in the gait cycle, while avoiding redundant calculation.

[0135] Figure 11 is a block diagram of an image spatio-temporal feature extraction device according to an example embodiment of the present application. As shown in Figure 11 The example image spatio-temporal feature extraction device 1100 includes:

[0136] An image embedding module 1110 is configured to obtain a sequence of image frames to be processed, and convert each image frame to be processed in the sequence of image frames to be processed into an embedding vector by using an embedding encoding layer. The sequence of image frames to be processed includes a plurality of image frames to be processed sorted in time sequence.

[0137] A spatial feature extraction module 1120 is configured to input the embedding vector into a spatial feature extraction layer to perform spatial feature extraction, and obtain a spatial feature vector corresponding to the sequence of image frames to be processed.

[0138] A temporal feature extraction module 1130 is configured to input the spatial feature vector into a temporal feature extraction layer to perform temporal feature extraction, and obtain a temporal feature vector corresponding to the sequence of image frames to be processed.

[0139] A feature fusion module 1140 is configured to fuse the spatial feature vector and the temporal feature vector by using a spatio-temporal feature fusion layer, and obtain a spatio-temporal feature vector corresponding to the sequence of image frames to be processed.

[0140] Figure 12 is a block diagram of a gait recognition device according to an example embodiment of the present application. As shown in Figure 12As shown, the example gait recognition apparatus 1200 includes:

[0141] An image acquisition module 1210, configured to acquire a gait image frame sequence obtained by image acquisition on a target object.

[0142] A feature extraction module 1220, configured to perform spatio-temporal feature extraction on the gait image frame sequence based on a spatio-temporal feature extraction network, to obtain a spatio-temporal feature vector corresponding to the target object in the gait image frame sequence.

[0143] A gait recognition module 1230, configured to perform gait classification on the target object based on the spatio-temporal feature vector, to obtain a gait recognition result corresponding to the target object.

[0144] It should be noted that the image spatio-temporal feature extraction apparatus and the gait recognition apparatus provided in the above embodiments belong to the same concept as the image spatio-temporal feature extraction method and the gait recognition method provided in the above embodiments, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. The image spatio-temporal feature extraction apparatus and the gait recognition apparatus provided in the above embodiments can allocate the above functions to different functional modules to complete all or part of the functions described above according to actual application, i.e., the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, which is not limited herein.

[0145] Please refer to Figure 13 , Figure 13 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 1300 includes a memory 1301 and a processor 1302, and the processor 1302 is configured to execute program instructions stored in the memory 1301 to implement the steps in any of the above image spatio-temporal feature extraction method or gait recognition method embodiments. In one specific implementation scenario, the electronic device 1300 can include but is not limited to: a microcomputer, a server, in addition, the electronic device 1300 can also include a notebook computer, a tablet computer and other mobile devices, which are not limited herein.

[0146] Specifically, the processor 1302 is configured to control itself and the memory 1301 to implement the steps in any of the above-described image spatio-temporal feature extraction method or gait recognition method embodiments. The processor 1302 can also be referred to as a central processing unit (CPU). The processor 1302 can be an integrated circuit chip including a processing core. The processor 1302 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 1302 can be implemented by an integrated circuit chip jointly working with other chips.

[0147] Referring to Figure 14 , Figure 14 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 1400 stores program instructions 1410 capable of being executed by a processor, and the program instructions 1410 are used to implement the steps in any of the above-described image spatio-temporal feature extraction method or gait recognition method embodiments.

[0148] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0149] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0151] In addition, each of the function units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A method for extracting spatiotemporal features from images, characterized in that, An application is made to a spatiotemporal feature extraction network, which includes an embedding coding layer, a spatial feature extraction layer, a temporal feature extraction layer, and a spatiotemporal feature fusion layer. The spatial feature extraction layer consists of multiple cascaded Transformer Blocks, and the temporal feature extraction layer consists of multiple cascaded Mamba Blocks. The method includes: A sequence of image frames to be processed is obtained, and each image frame in the sequence is converted into an embedding vector using the embedding coding layer; wherein, the sequence of image frames to be processed contains multiple image frames to be processed ordered in chronological order. The embedding vector is input to the spatial feature extraction layer to extract spatial features, thereby obtaining the spatial feature vector corresponding to the image frame sequence to be processed. The spatial feature vector is input into the temporal feature extraction layer to extract temporal features, thereby obtaining the temporal feature vector corresponding to the image frame sequence to be processed; The spatial feature vector and the temporal feature vector are fused using the spatiotemporal feature fusion layer to obtain the spatiotemporal feature vector corresponding to the image frame sequence to be processed.

2. The method according to claim 1, characterized in that, The step of converting each image frame in the sequence of image frames to be processed into an embedding vector using the embedding coding layer includes: Each image frame in the sequence of image frames to be processed is segmented into multiple image blocks; Embedding vector transformation is performed on each image block to generate an embedding vector for each image block.

3. The method according to claim 2, characterized in that, The embedding coding layer includes a three-dimensional linear projection layer and a multilayer perceptron; the step of using the embedding coding layer to convert each image frame in the image frame sequence into an embedding vector includes: The three-dimensional linear projection layer is used to segment each image frame to be processed into image blocks, and each image block is flattened and mapped to the channel dimension to obtain the initial vector corresponding to each image block. The initial vector corresponding to each image block is mapped using the multilayer perceptron to obtain the embedding vector corresponding to each image block.

4. The method according to claim 3, characterized in that, The method further includes: Extract image features from multiple modalities of each image frame to be processed to obtain a sequence of feature maps for multiple modalities; The process involves using the three-dimensional linear projection layer to segment each image frame into image blocks, flattening each image block, and mapping it to the channel dimension to obtain an initial vector corresponding to each image block, including: The feature map sequences of the multiple modalities are segmented into image blocks using the three-dimensional linear projection layer, and each image block is flattened and mapped to the channel dimension to obtain the initial vectors of the multiple modalities corresponding to each image block. The step of mapping the initial vector corresponding to each image patch using the multilayer perceptron to obtain the embedding vector corresponding to each image patch includes: The initial vectors of multiple modalities corresponding to each image block are concatenated to obtain the multimodal concatenation vector corresponding to each image block. The multilayer perceptron is used to map the multimodal stitching vector corresponding to each image block to obtain the embedding vector corresponding to each image block.

5. The method according to claim 2, characterized in that, The step of inputting the embedded vector into the spatial feature extraction layer to extract spatial features and obtain the spatial feature vector corresponding to the image frame sequence to be processed includes: The embedding vector of each image patch is input into the spatial feature extraction layer to extract spatial features, thereby obtaining the spatial feature vector corresponding to each image patch; The step of inputting the spatial feature vector into the temporal feature extraction layer for temporal feature extraction to obtain the temporal feature vector corresponding to the image frame sequence to be processed includes: The spatial feature vectors of image blocks belonging to the same image frame to be processed are concatenated to obtain the embedding vector of each image frame to be processed. The embedding vectors of each image frame to be processed are sorted according to the temporal order of each image frame to obtain the vector sorting result. The vector sorting result is input into the temporal feature extraction layer for temporal feature extraction to obtain the temporal feature vector corresponding to the image frame sequence to be processed.

6. A gait recognition method, characterized in that, The method includes: Obtain a sequence of gait image frames obtained from image acquisition of the target object; Based on the spatiotemporal feature extraction network according to any one of claims 1 to 5, spatiotemporal features are extracted from the gait image frame sequence to obtain the spatiotemporal feature vector corresponding to the target object in the gait image frame sequence; Based on the spatiotemporal feature vector, the target object is classified into gait types to obtain the gait recognition result corresponding to the target object.

7. The method according to claim 6, characterized in that, The method further includes: Extract image features of multiple modalities of the target object from the gait image frame sequence to obtain a feature map sequence of multiple modalities corresponding to the target object; The spatiotemporal feature extraction network based on any one of claims 1 to 5 performs spatiotemporal feature extraction on the gait image frame sequence to obtain the spatiotemporal feature vector corresponding to the target object in the gait image frame sequence, including: The feature map sequence of multiple modalities corresponding to the target object is input into the spatiotemporal feature extraction network for spatiotemporal feature extraction to obtain the spatiotemporal feature vector corresponding to the target object in the gait image frame sequence.

8. The method according to claim 7, characterized in that, The step of extracting image features of multiple modalities of the target object from the gait image frame sequence to obtain a feature map sequence of multiple modalities corresponding to the target object includes: Extract the gait feature map, and / or the movable part analytical feature map, and / or the key point feature map of the target object from the gait image frame sequence to obtain the gait feature map sequence, and / or the movable part analytical feature map sequence, and / or the key point feature map sequence corresponding to the target object in the gait image frame sequence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps in the image spatiotemporal feature extraction method as described in any one of claims 1-5 or the steps in the gait recognition method as described in any one of claims 6-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps in the image spatiotemporal feature extraction method as described in any one of claims 1-5 or the steps in the gait recognition method as described in any one of claims 6-8.

Images