A homomorphic heterogeneous data feature extraction method for gait recognition

Abstract

The application provides a homomorphism heterogeneous data feature extraction method for gait recognition, comprising: obtaining skeleton joint data and contour graph frames from a target walking video sequence, processing the skeleton joint data, performing division processing and morphology context extraction on contour point data based on the skeleton joint data, so as to form a contour graph data structure; performing feature extraction on the skeleton sequence to obtain posture features; performing feature extraction on the contour graph data structure to obtain contour features; and performing fusion processing on the posture features and the contour features to generate feature data for gait recognition, wherein the fusion processing is: multiplying the posture features and the contour features by corresponding weights respectively and then performing addition processing. The application adopts a combination of front-end fusion and back-end fusion, extracts homomorphism heterogeneous data features, and performs gait recognition based on the homomorphism heterogeneous data features, so that the accuracy of gait recognition can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gait recognition, in particular, especially relates to a homomorphism heterogeneous data feature extraction method for gait recognition. BACKGROUND

[0002] At present, most of the gait recognition techniques are researched from two directions. One method is to obtain the gait contour of the pedestrian by eliminating the background information from a video sequence, and to obtain the gait feature by using the dynamic appearance information contained in the contour sequence. However, the feature extracted by this method will change greatly when the gait state changes, for example, when the pedestrian carries an object with his hand or carelessly blocks the legs. Another method extracts the skeleton graph from the pedestrian video sequence, thereby eliminating the influence of external information on the human body shape and only retaining the more sensitive dynamic gait information, thereby having stronger robustness. However, this method cannot retain the static shape information of the person, such as the height and fatness information, and the recognition accuracy is not high in actual application. SUMMARY

[0003] In view of the technical problem of low gait recognition accuracy based on the existing method for extracting gait features, a homomorphism heterogeneous data feature extraction method is provided. The present application adopts a combination of front-end fusion and back-end fusion to extract homomorphism heterogeneous data features, and performs gait recognition based on the homomorphism heterogeneous data features, thereby improving the accuracy of gait recognition.

[0004] The technical means adopted by the present application are as follows:

[0005] A homomorphism heterogeneous data feature extraction method for gait recognition comprises:

[0006] Obtaining a target walking video sequence, obtaining skeleton joint data and contour frame from the target walking video sequence;

[0007] Processing the skeleton joint data to generate a skeleton sequence with the same number of frames as the contour frame and time alignment; performing equidistant sampling on the contour frame to generate contour point data, and performing division processing and shape context extraction on the contour point data based on the skeleton joint data, thereby forming a contour graph data structure;

[0008] Extracting features from the skeleton sequence to obtain posture features; extracting features from the contour graph data structure to obtain contour features;

[0009] The posture feature and the contour feature are fused to generate feature data for gait recognition, the fusion processing being: multiplying the posture feature and the contour feature by corresponding weights respectively and then adding them, the corresponding weights being obtained by using a sigmoid function after the splicing feature is processed by a full connection layer, the splicing feature being obtained after the posture feature and the contour feature are spliced.

[0010] Further, the contour point data is divided based on the skeleton joint data, and a shape context is extracted, to form a contour graph data structure, including:

[0011] The contour point data is divided based on the skeleton joint data, and a shape context is extracted for each region with a key point as the center, and then a contour graph data structure is formed by using the natural physical connection relationship of the key points.

[0012] Further, the posture feature is extracted from the skeleton sequence, including: processing the skeleton sequence based on a spatio-temporal graph convolution network to generate the posture feature, the spatio-temporal graph convolution network including ten stacked spatio-temporal convolution layers.

[0013] Further, the contour feature is extracted from the contour graph data structure, including: processing the contour graph data structure based on a spatio-temporal graph convolution network to generate the contour feature, the spatio-temporal graph convolution network including ten stacked spatio-temporal convolution layers.

[0014] Further, the ten stacked spatio-temporal convolution layers include:

[0015] The first spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the spatial feature of the human skeleton node is extracted, the size of the convolution kernel of the temporal graph convolution is 9, the time information between the sequences is extracted, the activation function is a Relu function, and the number of channels is changed from 3 to 64;

[0016] The second spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels is changed from 64 to 64;

[0017] The third spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels is changed from 64 to 64;

[0018] The fourth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels is changed from 64 to 64;

[0019] The fifth space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 64 to 128;

[0020] The sixth space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 128;

[0021] The seventh space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 128;

[0022] The eighth space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 256;

[0023] The ninth space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 256 to 256;

[0024] The tenth space-time graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the time graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 256 to 256.

[0025] Further, the weight corresponding to the posture feature and the weight corresponding to the contour feature are summed to 1.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] At present, most of the gait recognition technologies usually adopt two ways to recognize, one is to use the appearance information of the pedestrian, that is, the gait contour graph, but when the perspective is changed or the contour is changed greatly due to external factors, the robustness is poor, and the calculation amount of the contour sequence is also large, and the computer performance requirement is high. Two is to use the skeleton sequence of the pedestrian, which eliminates the influence of irrelevant variables, is more sensitive to dynamic information and event information, and has stronger robustness, and the disadvantage is that the effective appearance information of the pedestrian when walking is lost, such as height, fatness and the like. Therefore, the present algorithm improves the above-mentioned problems, adopts the combination of front-end fusion and back-end fusion, achieves the effect of improving the accuracy of gait recognition, and can be applied to safety sensitive places such as airports, banks, military security places and border ports, so as to guarantee the safety of the city and meet the stable development of the city and the safety needs of the public.

[0028] The application solves the data space misalignment problem by converting the common contour frame information to obtain the same contour graph structure as the skeleton data structure. And the way of converting the picture information into vectors reduces the amount of subsequent network processing. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] Figure 1 A homomorphic heterogeneous data feature extraction method flow chart for gait recognition.

[0031] Figure 2 A contour context information extraction flow chart.

[0032] Figure 3 A feature fusion flow chart. DETAILED DESCRIPTION

[0033] In order to make the person skilled in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0035] As Figure 1 shown, the present application provides a homomorphic heterogeneous data feature extraction method for gait recognition, comprising:

[0036] S1, acquire a target walking video sequence, obtain skeleton joint data and contour frame from the target walking video sequence.

[0037] In the present application, the openpose algorithm is used to directly estimate the coordinates of the skeleton joints and the confidence of each joint from the video sequence. By frame segmentation, background segmentation, binarization and morphological operation on the video sequence, a contour sequence is obtained.

[0038] S2, process the skeleton joint data to generate a skeleton sequence with the same number of frames as the contour frame and time alignment.

[0039] First, the skeleton joints are extracted from the pedestrian gait video sequence, and processed into a skeleton sequence with the same number of frames as the provided contour frame and time alignment.

[0040] S3, equally spaced sampling of the contour frame to generate contour point data, based on the skeleton joint data, the contour point data is divided and the morphological context is extracted, thereby forming a contour graph data structure.

[0041] The contour points are extracted from the contour graph using equally spaced sampling. The external contour is partitioned by selecting part of the key points that have a greater relationship with the gait from all the skeleton key points. For each region, the morphological context information of the region contour is extracted using an adaptive method, as shown in FIG. 1, and the natural physical connection relationship of the key points is used to form a contour graph data structure instead of the traditional image form. Figure 2

[0042] Specifically, according to the influence of the key points on the gait behavior and the stability of the key points. For example, the legs have the greatest influence on the gait, so multiple key points are selected for the legs. The external contour is partitioned based on human region segmentation method and demonstration test, and each key point is taken as the center, and the distance from the key point to the farthest point in the region is taken as the radius to obtain different circular regions. Each region contains only one key point, and the key point is used to extract the contour context information of the region. In the present application, the morphological context information refers to taking the key point as the center, drawing a circle with the distance from the key point to the farthest point in the region as the radius, then dividing the radius into n segments and the circular arc into m parts, and counting the number of contour points in each sector to obtain a vector to reflect the information of each point on the contour to the key point, which is called context information. Since the key points have the natural connection relationship of the human body, and the key points are selected based on human region division, the key points can be quickly connected according to the natural physical connection relationship. Thus, a contour graph structure is formed. The contour graph structure contains two parts: nodes and adjacency matrix. The nodes of the contour graph structure are the context contour features of each region key point, and the adjacency matrix is derived according to the natural connection of the key points.​

[0043] S4, feature extraction is performed on the skeleton sequence to obtain pose features, and feature extraction is performed on the contour graph data structure to obtain contour features.

[0044] In the application, the whole network utilizes the idea of deep learning and is mainly divided into two parts, namely a double-channel space-time graph convolution feature extraction network and a complementary learning fusion network.

[0045] The dual-channel spatio-temporal graph convolution is used to extract the features of each modality. Taking the graph convolution of the skeleton channel as an example, a spatio-temporal graph convolution layer includes a time graph convolution and a space graph convolution. The overall network is stacked by ten layers. The first layer of the space graph convolution has a convolution kernel size of 3, which is used to extract the spatial features of the human skeleton nodes. The time graph convolution has a convolution kernel of 9, which is used to extract the time information between sequences. The activation function is a Relu function. The spatio-temporal graph convolution first extracts the spatial features of the graph through the space graph convolution, then extracts the time information through the time graph convolution, and finally uses the residual network for output, with the channel number changing from 3 to 64. The second layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 64 to 64. The third layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 64 to 64. The fourth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 64 to 64. The fifth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 64 to 128. The sixth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 128 to 128. The seventh layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 128 to 128. The eighth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 128 to 256. The ninth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 256 to 256. The tenth layer is a spatio-temporal graph convolution layer mainly including a space graph convolution and a time graph convolution. The space graph convolution has a convolution kernel size of 3, the time graph convolution has a convolution kernel of 9, the activation function is a Relu function, and the channel number changes from 256 to 256. The structure of the other channel also adopts the same structure as described above.

[0046] S6, the posture feature and the contour feature are fused to generate feature data for gait recognition, the fusion processing is that the posture feature and the contour feature are multiplied by corresponding weights respectively and then added, the corresponding weights are obtained by using a sigmoid function after the splicing feature is processed through a full connection layer, and the splicing feature is obtained after the posture feature and the contour feature are spliced.

[0047] The two-channel features are fused by using a complementary learning module, and the network structure is as shown in Figure 3 The input is the shape, pose and splicing feature formed in series after the two-channel extracted features, a full connection layer is used to encode the features of two single modes to obtain an internal feature expression, then the splicing feature is processed through a full connection layer and a sigmoid function is used to obtain a score, and the score represents the contribution value of each dimension of the single mode to the total.

[0048] The score is a score for each dimension, since the sum of the contributions of the two modes to each dimension of the feature should be 1, the score of the other mode is 1-score, and the multiplication and addition of each feature and the corresponding score will obtain a fusion feature with complementary information. Specifically, the score in the present application is calculated by a network, and it corresponds to which mode is determined according to which mode is in front when the feature is spliced. The score is a (256x1) vector. Each dimension is a value between 0 and 1, and the (1-score) is used to obtain the score of the second mode,

[0049] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A homomorphic heterogeneous data feature extraction method for gait recognition, characterized in that, The method comprises the following steps: acquiring a target walking video sequence, obtaining skeleton joint data and contour frame from the target walking video sequence; processing the skeleton joint data to generate a skeleton sequence with the same number of frames as the contour frame and time alignment; performing equidistant sampling on the contour frame to generate contour point data, and performing regional processing and morphological context extraction on the contour point data based on the skeleton joint data, thereby forming a contour graph data structure; extracting posture features from the skeleton sequence; extracting contour features from the contour graph data structure; performing fusion processing on the posture features and the contour features to generate feature data for gait recognition, wherein the fusion processing is: multiplying the posture features and the contour features by corresponding weights and then adding them, and the corresponding weights are obtained by using a sigmoid function after concatenating the features through a fully connected layer.

2. The homomorphic heterogeneous data feature extraction method for gait recognition according to claim 1, characterized in that, The method comprises the following steps: performing regional processing and morphological context extraction on the contour point data based on the skeleton joint data, thereby forming a contour graph data structure, which comprises the following steps: 3.The method for homomorphic heterogeneous data feature extraction for gait recognition of claim 1, wherein, performing regional processing on the contour point based on the skeleton joint data, taking the key point as the center for each region, using an adaptive method to extract the morphological context information of the contour of the region, and then using the natural physical connection relationship of the key point to form a contour graph data structure.

4. The homomorphic heterogeneous data feature extraction method for gait recognition according to claim 1, characterized in that, The method comprises the following steps:

5. The homomorphic heterogeneous data feature extraction method for gait recognition according to claim 3 or 4, characterized in that, extracting posture features from the skeleton sequence, which comprises the following steps: processing the skeleton sequence based on a spatio-temporal graph convolution network to generate posture features, wherein the spatio-temporal graph convolution network comprises ten stacked spatio-temporal convolution layers. extracting contour features from the contour graph data structure, which comprises the following steps: processing the contour graph data structure based on a spatio-temporal graph convolution network to generate contour features, wherein the spatio-temporal graph convolution network comprises ten stacked spatio-temporal convolution layers. The ten stacked spatio-temporal convolution layers comprise the following layers: a first spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, which is used to extract the spatial features of the human skeleton node, the size of the convolution kernel of the temporal graph convolution is 9, which is used to extract the time information between the sequences, the activation function is a Relu function, and the number of channels changes from 3 to 64; a second spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 64 to 64; a third spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 64 to 64; a fourth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 64 to 64; a fifth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the size of the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 64 to 128; The sixth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 128; The seventh spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 128; The eighth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 128 to 256; The ninth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 256 to 256; The tenth spatio-temporal graph convolution layer, wherein the size of the convolution kernel of the spatial graph convolution is 3, the convolution kernel of the temporal graph convolution is 9, the activation function is a Relu function, and the number of channels changes from 256 to 256.

6. The homomorphic heterogeneous data feature extraction method for gait recognition according to claim 1, characterized in that, The weight corresponding to the posture feature and the weight corresponding to the contour feature are summed to 1.

Images