Gait recognition method from frontal perspective based on local fuzzy optical flow histogram energy map

Abstract

The present invention discloses a frontal perspective gait recognition method based on a local fuzzy optical flow histogram energy map. The present invention comprises the following steps: collecting frontal perspective gait video; calculating motion information of pixel points in the lower limb region of the human body; dividing the lower limb region of the human body into multiple local regions, modeling the gait motion information of the local regions of the human body, and obtaining a local fuzzy optical flow histogram energy map as a gait representation; using a histogram feature learning network to mine the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map, and mapping it to a discriminative deep feature space; utilizing the difference in deep features between the registered gait representation and the verified gait representation to realize frontal perspective gait recognition. The human gait representation proposed by the present invention has the characteristics of small dimension, strong robustness, simplicity and effectiveness; the present invention effectively realizes nonlinear mapping from histogram features to deep features, which helps to improve the accuracy of frontal perspective gait recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a frontal perspective gait recognition method, in particular to a frontal perspective gait recognition method based on a local fuzzy optical flow histogram energy map. Background Art

[0002] Gait recognition aims to identify or verify the identity of pedestrians by analyzing and extracting the gait motion characteristics of pedestrians. It is an important topic in the field of pattern recognition and biometric recognition. Gait recognition, as a new generation of biometric recognition technology, is widely used in video surveillance and other fields due to its non-contact, concealed and long-distance perceptibility. Since the gait motion from the side view is easy to obtain and the spatiotemporal information is intuitive, most of the current gait recognition work is based on the gait motion from the side view. However, in actual video surveillance applications, especially in security-sensitive environments, surveillance cameras are usually placed in narrow corridors, entrances and exits, and the data collection area is small. What is often captured is the gait motion of individuals from the front view. The size of the moving individual from the front view will change with the distance from the camera, and the spatiotemporal motion characteristics of the gait are not obvious. The recognition accuracy of the gait recognition method based on the side view will be greatly reduced or even fail. To promote the practical application of gait recognition technology, gait recognition from the front view is an issue that cannot be avoided.

[0003] At present, the research work on frontal perspective gait recognition is mainly divided into methods based on depth cameras and methods based on RGB cameras. Although these methods achieve a high correct recognition rate in frontal perspective gait recognition, there are the following problems: (1) Most methods based on RGB cameras usually extract human gait contour images from RGB gait videos and extract gait spatiotemporal information from the changes in human contour shapes. This requires cumbersome image preprocessing and precise human contour extraction and alignment technology, and the recognition accuracy also depends on the quality of the gait contour image. (2) Many methods based on depth cameras use the depth change information of pedestrians to more comprehensively describe the gait movement, or derive human skeletal joint data for human modeling and feature extraction. Although this can make up for the shortcomings of RGB cameras and provide more comprehensive gait information, it usually requires a huge amount of computation, and the recognition accuracy depends on the posture estimation technology. In addition, it is not realistic to deploy and use expensive depth cameras in surveillance environments. In fact, since human gait movement is a complex nonlinear movement, and each body part has its own time-varying dynamic pattern, if the gait movement information of each local part of the human body can be accurately described and extracted based on RGB gait video information, and the gait movement characteristics can be extracted and learned, it will obviously be of great help to the actual application of frontal perspective gait recognition. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing methods and provide a frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map, which is a simpler and more effective frontal perspective gait recognition method with strong feature robustness and high recognition accuracy.

[0005] The present invention directly calculates the motion information of pixels on the frontal perspective gait video to obtain the gait motion characteristics of the local parts of the lower limbs of the human body, and models the gait motion information through the local fuzzy optical flow histogram energy map as the human gait representation. A histogram feature learning network is constructed, and the histogram feature learning network is used to further explore the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map, and mapped to a discriminative deep feature space to extract a discriminative deep feature vector. Finally, by calculating the difference between the registered gait representation and the verified gait representation in the deep feature vector, the identity of the registered gait representation with the smallest distance to the verified gait representation is returned as the identity recognition result, thereby realizing frontal perspective gait recognition.

[0006] The specific technical solution of the present invention is achieved through the following steps:

[0007] Step 1: Collect gait video from the frontal perspective;

[0008] Collect gait videos from the frontal perspective and obtain the minimum circumscribed rectangular area of ​​the human body in each frame of gait images;

[0009] Step 2: Calculate the motion information of the pixels in the lower limb area of ​​the human body;

[0010] The lower limb area of ​​the human body is defined as the area of ​​interest; the pixel point motion of the lower limb area of ​​the human body in two adjacent frames of gait images is calculated to construct an optical flow field describing the gait motion information;

[0011] Step 3: extract the local fuzzy optical flow histogram energy map;

[0012] The lower limb area of ​​the human body is divided into six local areas, representing the left thigh, right thigh, left calf, right calf, left foot, and right foot areas respectively. At the same time, the optical flow field of each local area of ​​the lower limb area is obtained by dividing the optical flow field of the lower limb area to describe the gait motion information of the corresponding local parts;

[0013] In a gait image sequence, the optical flow directions describing the gait motion characteristics are counted according to the optical flow map of the local area of ​​the lower limbs of the human body, and the local fuzzy optical flow direction histogram is calculated; the local fuzzy optical flow direction histogram is averaged in the time dimension, and the local fuzzy optical flow histogram energy map is extracted as the human gait representation;

[0014] Step 4: Histogram feature learning and gait recognition;

[0015] Construct a histogram feature learning network; use the histogram feature learning network to mine the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map, and map it to a discriminative deep feature space to extract a discriminative deep feature vector; calculate the difference between the registered gait representation and the verified gait representation in the deep feature vector, and return the identity of the registered gait representation with the smallest distance to the verified gait representation as the identity recognition result, thereby realizing gait recognition from the frontal perspective.

[0016] Furthermore, in step 1, a human gait video from a frontal perspective is collected to obtain an original gait image B of each frame. t The minimum circumscribed rectangular area of ​​the human body is denoted as B t (x ul ,y ul , x lr ,y lr ), t = 1, 2, ..., T represents the sequential index of the gait image, and T is the length of the gait image sequence in a gait video; (x ul ,y ul ) and (x lr ,y lr ) are the two-dimensional coordinates of the upper left corner and the lower right corner of the minimum circumscribed rectangular area of ​​the human body; the height of the minimum circumscribed rectangular area of ​​the human body is H=x lr -x ul , width is W = y lr -y ul .

[0017] Further, in step 2, the region of interest is a human lower limb region; the human lower limb region can be expressed as:

[0018]

[0019] Among them, S t is the lower limb area of ​​the human body; the two-dimensional coordinates of the upper left corner of the lower limb area of ​​the human body are The 2D coordinates of the lower right corner are The height of the lower limb area of ​​the human body is Width is

[0020] Further, in step 2, the gait motion information of the lower limb region of the human body is described by the optical flow field of the lower limb region of the human body in two adjacent frames of gait images; the optical flow field of the lower limb region of the human body is the pixel point motion of the lower limb region of the human body in two adjacent frames of gait images; the optical flow field v t The mathematical representation of is:

[0021] v t={[ρ(x, y), θ(x, y)]|x=1, 2,..., H′; y=1, 2,..., W′; (x, y)∈S t} (2)

[0022] Among them, ρ(x, y) and θ(x, y) are the optical flow amplitude and optical flow direction of the pixel with coordinates (x, y), respectively.

[0023] Further, in step 3, the method for dividing the lower limb region of the human body is: dividing the lower limb region of the human body into six local regions, representing the right thigh, left thigh, right calf, left calf, right foot and left foot regions respectively; the six local regions are represented as:

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in They are the right thigh, left thigh, right calf, left calf, right foot and left foot areas.

[0031] Furthermore, in step 2, the method for dividing the optical flow field of the lower limb region of the human body is: according to the method for dividing the lower limb region of the human body in step 3, the optical flow field of the lower limb region of the human body is divided into six sub-optical flow fields of local parts; the mathematical expression of the optical flow fields of the six local parts is:

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] in Represents the optical flow fields of the right thigh, left thigh, right calf, left calf, right foot and left foot areas respectively.

[0039] Furthermore, in step three, the gait motion characteristics are described by the optical flow information of the local part of the lower limbs of the human body, and the local gait motion characteristics are modeled by the local fuzzy optical flow histogram energy map; the calculation method of the local fuzzy optical flow histogram energy map is as follows:

[0040] (1) Calculate the local optical flow direction histogram:

[0041] (1.1) Calculate the optical flow direction histogram of each local area of ​​each gait image:

[0042]

[0043]

[0044]

[0045] in is the optical flow direction histogram of the i-th local area of ​​the t-th frame gait image, i = 1, 2, ..., 6 represents the index of the local part of the human lower limb; B is the number of histogram bins, indicating that the given angle range is evenly divided into B parts; h j is the jth histogram bin, j th bin represents the jth histogram bin; τ is the optical flow amplitude threshold, which means that when the optical flow amplitude ρ(x, y) is greater than or equal to τ, the corresponding optical flow direction θ(x, y) can be effectively counted;

[0046] (1.2) Concatenate the optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional optical flow direction histogram of the i-th local area:

[0047]

[0048] in[·] C Represents the splicing operation in the time dimension, is the two-dimensional optical flow direction histogram of the i-th local area;

[0049] (1.3) Concatenate the two-dimensional optical flow direction histograms of each local area in the horizontal dimension and calculate the local optical flow direction histogram:

[0050] H=[H i |i=1,2,...6] W , (19)

[0051] in[·] W Represents the splicing operation performed in the horizontal dimension, is the local optical flow direction histogram;

[0052] (2) Calculate the local blurred optical flow direction histogram:

[0053] (2.1) Define the fuzzy optical flow direction histogram of each local area of ​​each gait image:

[0054]

[0055]

[0056]

[0057] in is the fuzzy optical flow direction histogram of the i-th local area of ​​the t-th frame gait image; X×Y is the image resolution of the local part; μ jk is the optical flow direction θ of the kth pixel k The membership degree of the jth histogram bin is used to flexibly allocate the optical flow direction to all associated histogram bins, ρ k Represents the optical flow amplitude of the kth pixel;

[0058] (2.2) Define the fuzzy membership function:

[0059]

[0060] Among them, μ jk is the optical flow direction θ of the kth pixel k The degree of membership with the jth histogram bin; φ j is the center point of the jth histogram bin; φ l is the center point of the lth histogram bin; m is the fuzzy coefficient, which is used to control the similarity between different histogram bins; σ is the variance of the Gaussian term, which is used to control the influence between adjacent histogram bins during the fuzzification process;

[0061] (2.3) First, based on step (1.1), create an optical flow direction histogram with B′ histogram bins, denoted as a fine histogram, and assign angles in the range of [-π, π) to the fine histogram; create a common histogram with B histogram bins, denoted as a coarse histogram, where B′>B; based on the fuzzy membership function described in step (2.2), use the fuzzy c-means clustering algorithm to classify all fine histogram bins into B classes, where each class represents a coarse histogram bin, and obtain the membership of all fine histogram bins to each coarse histogram bin, and express it as a membership matrix:

[0062] M(B; B′; μ(m; σ)) = [m pq ] B×B′ (twenty four)

[0063] where μ(m;σ) is the fuzzy membership function in step (2.2); M is the membership matrix, where m pqis the membership of the qth fine histogram bin to the pth coarse histogram bin. Since the number of coarse histogram bins is the same as the number of histogram bins of the fuzzy optical flow direction histogram described in step (2.1), each class can represent a fuzzy optical flow direction histogram bin, so when p = j and the optical flow direction θ k When it belongs to the qth fine histogram bin, m pq =μ jk ;

[0064] (2.4) Based on step (1.1), calculate the optical flow direction histogram of each local area of ​​each gait image The number of histogram bins is B′; based on the membership matrix obtained in step (2.3), the fuzzy optical flow direction histogram of each local area of ​​each gait image in step (2.1) is calculated by the following formula:

[0065]

[0066] (2.5) Concatenate the fuzzy optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional fuzzy optical flow direction histogram of the i-th local area:

[0067]

[0068] in is the two-dimensional fuzzy optical flow direction histogram of the i-th local area;

[0069] (2.6) The two-dimensional fuzzy optical flow direction histograms of each local area are spliced ​​in the horizontal dimension to calculate the local fuzzy optical flow direction histogram:

[0070] F=[F i |i=1,2,…6] W (27)

[0071] in is the local blurred optical flow direction histogram;

[0072] (3) Perform an average operation on the local fuzzy optical flow direction histogram in the spatial dimension and calculate the local fuzzy optical flow histogram energy map:

[0073]

[0074] in is the local blurred optical flow histogram energy map.

[0075] Furthermore, in step four, the histogram feature learning network uses the local fuzzy optical flow histogram energy map as network input, mines the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map by training the network, maps the histogram features to a discriminative deep feature space, and outputs a discriminative deep feature vector.

[0076] Further, in step 4, the histogram feature learning network includes a nonlinear enhancement module, a feature extraction module and a downsampling module connected in sequence;

[0077] The nonlinear enhancement module includes two nonlinear enhancement layers connected in sequence, each nonlinear enhancement layer includes a 1×1 convolution layer, a batch normalization layer and a linear rectification function connected in sequence, wherein the 1×1 convolution layer is used to change the number of feature maps to enhance the nonlinear expression ability of features;

[0078] The feature extraction module includes a plurality of dense connection modules, wherein the dense connection modules realize external dense connection with each other, and the input of any dense connection module is the splicing result of the output of all the previous dense connection modules and the output of the nonlinear enhancement module in the channel dimension; the dense connection module includes a plurality of groups of dense convolutional layers, wherein the dense convolutional layers realize internal dense connection with each other, and the input of any dense convolutional layer is the splicing result of the output of all the previous dense convolutional layers and the original input of the dense connection module in the channel dimension; the dense convolutional layer includes a batch normalization layer, a linear rectification function, a 1×1 convolutional layer, a batch normalization layer, a linear rectification function and a 3×3 convolutional layer connected in sequence;

[0079] The downsampling module includes two parallel connected pooling layers, the two pooling layers are a 3×3 adaptive maximum pooling layer and a 3×3 adaptive average pooling layer respectively; the downsampling module takes the output of the feature extraction module as input, performs dimensionality reduction processing on the input, and outputs two sets of feature maps with the same dimensional size;

[0080] The output deep features of the histogram feature learning network are the concatenation of the output features of the downsampling modules and are expanded into a one-dimensional feature vector;

[0081] Furthermore, in step four, the gait recognition method calculates the difference between the registered gait representation and the verified gait representation in the deep feature vector in a discriminative deep feature space, returns the identity of the registered gait representation with the smallest distance from the verified gait representation as the identity recognition result, and realizes gait recognition from the frontal perspective.

[0082] Compared with the existing frontal perspective gait recognition solutions, the invention has the following advantages:

[0083] 1. The present invention directly calculates the motion information of pixels on the frontal perspective gait video to obtain the gait motion information of the local parts of the lower limbs of the human body, and models the local gait motion information through the local fuzzy optical flow histogram energy map. The histogram feature learning network is further used to mine the intrinsic gait spatiotemporal characteristics of the local fuzzy optical flow histogram energy map, and the nonlinear expression and discrimination capabilities of the gait features are enhanced, so that it can better reflect the complex nonlinear motion pattern of the human gait, thereby achieving gait recognition with high recognition accuracy and strong robustness.

[0084] 2. The local fuzzy optical flow histogram energy map extracted by the present invention can accurately model the pixel point motion of the human gait image, accurately reflect the local parts of the human lower limbs and the dynamic mode of the gait system of the human lower limbs, and has the characteristics of small dimension, strong robustness and simple calculation.

[0085] 3. Compared with the deep learning method, the local fuzzy optical flow histogram energy map features lack semantic features such as pixels, textures and contours due to the different input methods such as images and videos. The present invention designs a novel histogram feature learning network to replace the traditional convolutional neural network for deep feature extraction, which can be extended to pattern recognition tasks based on histogram features. The histogram feature network first introduces a nonlinear enhancement module, which gradually increases the number of feature maps through two layers of 1×1 convolutional layers to retain and enhance the nonlinear gait motion characteristics inherent in the histogram feature as much as possible. In addition, the dense connection module is improved, the traditional linear connection and transition layer are abandoned, and external dense connections are used to realize the interconnection between dense connection modules, further strengthening the propagation of low-resolution and weakly expressive histogram features in the network and promoting feature reuse, effectively realizing the nonlinear mapping from histogram features to deep features, which helps to improve the accuracy of gait recognition from the frontal perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a flow chart of a frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map proposed by the present invention.

[0087] Figure 2 Schematic diagram of the method for dividing the lower limb area of ​​the human body.

[0088] Figure 3 is a schematic diagram of the external density achieved between densely connected modules.

[0089] Figure 4 Detailed structure of the histogram feature learning network used in the embodiment.

[0090] Figure 5 It is the recognition accuracy of the local fuzzy optical flow direction histogram under different setting parameters in the embodiment. DETAILED DESCRIPTION

[0091] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0092] The present invention discloses a frontal perspective gait recognition method based on a local fuzzy optical flow histogram energy map, comprising the following steps: collecting a frontal perspective gait video; calculating motion information of pixel points in a lower limb region of a human body; dividing the lower limb region of a human body into a plurality of local regions, modeling the gait motion information of the local regions of the human body, and obtaining a local fuzzy optical flow histogram energy map as a gait representation; using a histogram feature learning network to mine the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map, and mapping it to a discriminative deep feature space; and utilizing the difference in deep features between a registered gait sequence and a verification sample sequence to realize frontal perspective gait recognition. The present invention directly extracts the spatiotemporal variation pattern of gait motion from human gait video, avoiding complex image preprocessing process; the extracted local fuzzy optical flow histogram energy map can accurately model the gait motion characteristics of human body, and has the characteristics of small dimension, strong robustness, simplicity and effectiveness; a histogram feature learning network is used to replace the traditional convolutional neural network, and a nonlinear enhancement module is introduced to retain and enhance the nonlinear gait motion characteristics inherent in the histogram feature as much as possible, and multiple dense connection modules are used for deep feature extraction, wherein the dense connection modules realize external dense connection instead of traditional linear connection, strengthen the propagation of low-resolution and weakly expressive histogram features in the network and promote feature reuse, which is helpful to improve the accuracy of gait recognition from the frontal perspective.

[0093] Example 1

[0094] like Figure 1 As shown, the frontal perspective gait recognition method based on the local fuzzy optical flow histogram energy map specifically includes the following steps:

[0095] Step 1: Collect human gait video from the frontal perspective;

[0096] The gait database used in this embodiment is the CASIA-B gait database. The CASIA-B gait database contains 124 people in 11 different viewing angles (0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, 180°) and three different walking states (normal walking state (NM), wearing a coat walking state (CL), carrying a backpack walking state (BG)), wherein each person has 4 gait sequences of normal walking state (NM#01-06), 2 gait sequences of carrying a backpack walking state (CL#01-02) and 2 gait sequences of carrying a backpack walking state (BG#01-02) in each viewing angle, a total of 124×11×(6+2+2)=13640 gait videos. In order to meet the research purpose of the present invention, this embodiment selects all gait videos of 124 people under the frontal perspective (0°), a total of 124×(6+2+2)=1240 gait videos as research objects. The gait videos of the first 74 people under all the frontal perspectives are selected as the training set, and the gait videos of the remaining 50 people are used for testing. In the test phase, the first four gait sequences of normal walking state (NM#01-04) are used to register the gait database, and the remaining two gait sequences of normal walking state (NM#05-06), two gait sequences of walking state with backpacks (CL#01-02) and two gait sequences of walking state with backpacks (BG#01-02) are used as verification samples respectively.

[0097] The present invention obtains the minimum bounding rectangle area of ​​the human body in each frame of the gait image for all gait videos used. The minimum bounding rectangle area of ​​the human body is denoted as B t (x ul ,y ul , x lr ,y lr ), where B t is a gait image, t=1, 2, ..., T represents the sequential index of the gait image, and T is the number of image frames in a gait video; (x ul ,y ul ) and (x lr ,y lr ) are the two-dimensional coordinates of the upper left corner and the lower right corner of the minimum circumscribed rectangular area of ​​the human body; the height of the minimum circumscribed rectangular area of ​​the human body is H=x lr -x ul , width is W = y lr -y ul .

[0098] Step 2: Calculate the motion information of the pixels in the lower limb area of ​​the human body;

[0099] First, define the lower limb area of ​​the human body as the region of interest and express it as the following formula:

[0100]

[0101] Where S t represents the lower limb area of ​​the human body, and the two-dimensional coordinates of the upper left corner of the lower limb area of ​​the human body are The 2D coordinates of the lower right corner are The height of the lower limb area of ​​the human body is Width is

[0102] Then, an optical flow field describing gait motion information is constructed by calculating the pixel point motion of the lower limb region of two adjacent frames of gait images in the gait image sequence. The mathematical representation of the optical flow field of the lower limb region of the human body is:

[0103] v t ={[ρ(x, y), θ(x, y)]|x=1, 2,..., W′; y=1, 2,..., H′; (x, y)∈S t}, where ρ(x, y) and θ(x, y) are the optical flow amplitude and direction of the pixel with coordinates (x, y), respectively.

[0104] Step 3: extract the local fuzzy optical flow histogram energy map;

[0105] First, the lower limb area of ​​the human body is divided into six local sub-areas, representing the left thigh, right thigh, left calf, right calf, left foot, and right foot areas, respectively. Their mathematical expressions are:

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] in They are the right thigh, left thigh, right calf, left calf, right foot and left foot areas.

[0113] Then, according to the above-mentioned method for dividing the lower limb region of the human body, the optical flow field of each local region of the lower limb is obtained by dividing the optical flow field of the lower limb region to describe the gait motion information of the corresponding local parts. The optical flow fields of the six local parts of the lower limb of the human body are expressed as follows:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] in The optical flow fields are respectively for the right thigh, left thigh, right calf, left calf, right foot and left foot.

[0121] Furthermore, in a gait image sequence, according to the optical flow map of the local area of ​​the lower limbs of the human body extracted above, the optical flow direction describing the gait motion characteristics is counted, and the local fuzzy optical flow direction histogram is calculated; the local fuzzy optical flow direction histogram is averaged in the time dimension, and the local fuzzy optical flow histogram energy map is extracted to model the human gait motion characteristics as a human gait representation. The calculation method of the local fuzzy optical flow histogram energy map is:

[0122] (1) Calculate the local optical flow direction histogram:

[0123] (1.1) Calculate the optical flow direction histogram of each local area of ​​each gait image:

[0124]

[0125]

[0126]

[0127] in is the optical flow direction histogram of the i-th local area of ​​the t-th frame gait image, i = 1, 2, ..., 6 represents the index of the local part of the human lower limb; B is the number of histogram bins, indicating that the angle range [-π, π) is evenly divided into B parts; h j is the jth histogram bin, j th bin represents the jth histogram bin; τ is the optical flow amplitude threshold, which means that when the optical flow amplitude ρ(x, y) is greater than or equal to τ, the corresponding optical flow direction θ(x, y) can be effectively counted.

[0128] (1.2) Concatenate the optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional optical flow direction histogram of the i-th local area:

[0129]

[0130] in[·] C Represents the splicing operation in the time dimension, is the two-dimensional optical flow direction histogram of the i-th local area.

[0131] (1.3) Concatenate the two-dimensional optical flow direction histograms of each local area in the horizontal dimension and calculate the local optical flow direction histogram:

[0132] H=[H i |i=1,2,...6] W ,

[0133] in[·] W Represents the splicing operation performed in the horizontal dimension, is the local optical flow direction histogram.

[0134] (2) Calculate the local blurred optical flow direction histogram:

[0135] (2.1) Define the fuzzy optical flow direction histogram of each local area of ​​each gait image:

[0136]

[0137]

[0138]

[0139] in is the fuzzy optical flow direction histogram of the i-th local area of ​​the t-th frame gait image; X×Y is the image resolution of the local part; μ jk is the optical flow direction θ of the kth pixel k The membership degree of the jth histogram bin enables flexible allocation of optical flow directions to all associated histogram bins; ρ k Represents the optical flow amplitude of the kth pixel.

[0140] (2.2) Define the fuzzy membership function:

[0141]

[0142] where μ jk is the optical flow direction θ of the kth pixel k The degree of membership with the jth histogram bin; φ j is the center point of the jth histogram bin; m is the fuzzy coefficient, which is used to control the similarity between different histogram bins; σ is the variance of the Gaussian term, which is used to control the influence between adjacent histogram bins during the fuzzification process.

[0143] (2.3) First, based on step (1.1), create an optical flow direction histogram with B′ bins, denoted as a fine histogram, and assign the angles in the range of [-π, π) to the fine histogram; create a normal histogram with B bins, denoted as a coarse histogram, where B′>B; based on the fuzzy membership function described in step (2.2), use the fuzzy c-means clustering algorithm to classify all fine histogram bins into B classes, where each class represents a coarse histogram bin, and obtain the membership of all fine histogram bins to each coarse histogram bin, and express it as the membership matrix obtained:

[0144] M(B; B′; μ(m; σ)) = [m pq ] B×H′ ,

[0145] where μ(m;σ) is the fuzzy membership function in step (2.2); M is the membership matrix, where m pq is the membership of the qth fine histogram bin to the pth coarse histogram bin. Since the number of coarse histogram bins is the same as the number of histogram bins of the fuzzy optical flow direction histogram described in step (2.1), each class can represent a fuzzy optical flow direction histogram bin, so when p = j and the optical flow direction θ k When it belongs to the qth fine histogram bin, m pq =μ jk .

[0146] (2.4) Based on step (1.1), calculate the optical flow direction histogram of each local area of ​​each gait image The number of histogram bins is B′; based on the membership matrix obtained in step (2.3), the fuzzy optical flow direction histogram of each local area of ​​each gait image in step (2.1) can be calculated by the following formula:

[0147]

[0148] (2.5) Concatenate the fuzzy optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional fuzzy optical flow direction histogram of the i-th local area:

[0149]

[0150] in is the two-dimensional fuzzy optical flow direction histogram of the i-th local area.

[0151] (2.6) The two-dimensional fuzzy optical flow direction histograms of each local area are spliced ​​in the horizontal dimension to calculate the local fuzzy optical flow direction histogram:

[0152] F=[F i |i=1,2,…6] W ,

[0153] in is the local blurred optical flow direction histogram.

[0154] (3) Perform an average operation on the local fuzzy optical flow direction histogram in the spatial dimension and calculate the local fuzzy optical flow histogram energy map:

[0155]

[0156] in is the local blurred optical flow histogram energy map.

[0157] Step 4: Histogram feature learning and gait recognition;

[0158] First, a histogram feature learning network is constructed, which includes a nonlinear enhancement module, a feature extraction module and a downsampling module connected in sequence.

[0159] The nonlinear enhancement module includes two nonlinear enhancement layers connected in sequence, each of which includes a 1×1 convolutional layer, a batch normalization layer and a linear rectification function connected in sequence, where the 1×1 convolutional layer is used to change the number of feature maps and enhance the nonlinear expression ability of the features.

[0160] The feature extraction module includes a plurality of dense connection modules, wherein the dense connection modules realize external dense connection with each other, and the input of any dense connection module is the splicing result of the output of all the previous dense connection modules and the output of the nonlinear enhancement module in the channel dimension; the dense connection module includes a plurality of groups of dense convolutional layers, wherein the dense convolutional layers realize internal dense connection with each other, and the input of any dense convolutional layer is the splicing result of the output of all the previous dense convolutional layers and the original input of the dense connection module in the channel dimension. The dense convolutional layer includes a batch normalization layer, a linear rectification function, a 1×1 convolutional layer, a batch normalization layer, a linear rectification function and a 3×3 convolutional layer connected in sequence.

[0161] Furthermore, the feature extraction module includes a dense connection module I, a dense connection module II, a dense connection module III and a dense connection module IV, wherein the dense connection module I takes the output of the nonlinear enhancement module as input, and any subsequent dense connection module takes the concatenation result of the output of all its previous dense connection modules and the output of the nonlinear enhancement module in the channel dimension as input, and the number of output feature maps of the dense connection module I, the dense connection module II, the dense connection module III and the dense connection module IV is k, and k is a pre-set positive integer. The dense connection module I, the dense connection module II, the dense connection module III and the dense connection module IV respectively include 4, 6, 6, and 8 groups of dense convolutional layers. For a dense connection module, any dense convolutional layer takes the concatenation result of the output of all its previous dense convolutional layers and the original input of the dense connection module in the channel dimension as input, and outputs k feature maps.

[0162] The downsampling module includes two parallel connected pooling layers, the two pooling layers are a 3×3 adaptive maximum pooling layer and a 3×3 adaptive average pooling layer respectively; the downsampling module takes the output of the feature extraction module as input, performs dimensionality reduction processing on the input, and outputs two sets of feature maps with the same dimensional size.

[0163] The output deep features of the histogram feature learning network are the concatenation of the output features of the downsampling module and are expanded into a one-dimensional feature vector.

[0164] For the gait video to be verified, the local fuzzy optical flow histogram energy map is extracted as the input of the above-mentioned trained histogram feature learning network, and the deep features output by the network are used to represent the individual walking pattern. By calculating the difference in deep features between the registered gait pattern and the verified gait pattern, the identity of the registered gait pattern with the smallest distance to the verified gait pattern is returned as the identity recognition result, thereby realizing gait recognition from the frontal perspective.

[0165] Since the local fuzzy optical flow direction histogram and the local fuzzy optical flow histogram energy map are both derived based on the local optical flow direction histogram, the present invention implements gait recognition by performing histogram comparison based on the chi-square comparison method to verify the method for dividing the lower limb area of ​​the human body. For the convenience of description, the method for dividing the lower limb area of ​​the human body of the present invention is denoted as S {1 ,2,3,4,5,6} , which is divided into the right thigh S 1 , left thigh S 2 , right calf S 3 , left calf S 4 , right foot S 5 and left foot S 6 In addition, use the thigh area S 7 , calf area S 8 , foot area S 9 , and lower limb area S10 As shown in Table 1, the recognition rate (Rank-1) results of the recognition experiment using the local optical flow direction histogram under different human lower limb area division methods are shown in the present invention. The experimental results show the effectiveness and superiority of the human lower limb area division method described in the present invention, which is helpful to extract the intrinsic motion characteristics of each local part of the human lower limb.

[0166] Table 1

[0167]

[0168]

[0169] The local fuzzy optical flow histogram energy map is derived by averaging the local fuzzy optical flow direction histogram in the time dimension. Based on the fuzzy membership function μ(m; σ) described in step (2.2), the membership of the fine histogram bins (the number of histogram bins is B') and the coarse histogram bins (the number of histogram bins is B) is calculated by using the fuzzy c-means clustering algorithm to derive the local fuzzy optical flow direction histogram. In order to obtain the optimal feature expression capability, this embodiment traverses different parameters (B', B, m, σ) to derive the optimal local fuzzy optical flow direction histogram. In this embodiment, gait recognition is achieved by performing histogram comparison based on the chi-square comparison method. As Figure 5 The experimental results show that when (B', B, m, σ) = (30, 24, 1.4, 18), the highest recognition rate of 68.5% can be obtained.

[0170] Using the human lower limb segmentation method and optimal parameters, the optimal local fuzzy optical flow histogram energy map of all gait videos is calculated as a representation of human gait. As shown in Table 2, the recognition rate of the frontal perspective gait recognition experiment using the deep features mapped by the local fuzzy optical flow histogram energy map in the present invention is shown. The experimental results prove the feasibility and superiority of the present invention. The method of the present invention can achieve a high recognition rate in frontal perspective gait recognition, and the recognition rate remains at a high level when the walking state changes, and has strong robustness to changes in clothing and carrying state.

[0171] Table 2

[0172]

[0173] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map, characterized in that: The following steps are included: Step 1: Collect gait video from the frontal perspective; Collect gait videos from the frontal perspective and obtain the minimum circumscribed rectangular area of ​​the human body in each frame of gait images; Step 2: Calculate the motion information of the pixels in the lower limb area of ​​the human body; Define the lower limb area of ​​the human body as the region of interest; Calculate the pixel motion of the lower limb area of ​​two adjacent frames of gait images and construct an optical flow field that describes the gait motion information; Step 3: extract the local fuzzy optical flow histogram energy map; The lower limb area of ​​the human body is divided into six local areas, representing the left thigh, right thigh, left calf, right calf, left foot, and right foot areas respectively. At the same time, the optical flow field of each local area of ​​the lower limb area is obtained by dividing the optical flow field of the lower limb area to describe the gait motion information of the corresponding local parts; In a gait image sequence, the optical flow directions describing the gait motion characteristics are counted according to the optical flow map of the local area of ​​the lower limbs of the human body, and the local fuzzy optical flow direction histogram is calculated; the local fuzzy optical flow direction histogram is averaged in the time dimension, and the local fuzzy optical flow histogram energy map is extracted as the human gait representation; Step 4: Histogram feature learning and gait recognition; Construct a histogram feature learning network; A histogram feature learning network is used to mine the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map and map it to a discriminative deep feature space to extract a discriminative deep feature vector. The difference between the registered gait representation and the verified gait representation in the deep feature vector is calculated, and the identity of the registered gait representation with the smallest distance to the verified gait representation is returned as the identity recognition result to achieve frontal perspective gait recognition.

2. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 1 is characterized in that: In step 1, a human gait video is collected from the front view to obtain the original gait image B of each frame. t The minimum circumscribed rectangular area of ​​the human body is denoted as B t (x ul ,y ul ,x lr ,y lr ), t=1,2,…,T represents the sequential index of gait images, and T is the length of the gait image sequence in a gait video; (x ul ,y ul ) and (x lr ,y lr ) are the two-dimensional coordinates of the upper left corner and the lower right corner of the minimum circumscribed rectangular area of ​​the human body; the height of the minimum circumscribed rectangular area of ​​the human body is H=y lr -y ul , the width is W = x lr -x ul .

3. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 1 or 2, characterized in that: In step 2, the region of interest is the lower limb region of the human body; the lower limb region of the human body can be expressed as: Among them, S t is the lower limb area of ​​the human body; the two-dimensional coordinates of the upper left corner of the lower limb area of ​​the human body are The 2D coordinates of the lower right corner are The height of the lower limb area of ​​the human body is Width is 4. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 3 is characterized in that: In step 2, the gait motion information of the lower limb region of the human body is described by the optical flow field of the lower limb region of the human body in two adjacent frames of gait images; the optical flow field of the lower limb region of the human body is the pixel point motion of the lower limb region of the human body in two adjacent frames of gait images; the optical flow field v t The mathematical representation of is: v t {[[ρ(x,y),θ(x,y)]|x=1,2,…,W;y=1,2,…,H;(x,y)∈S t } (2) Among them, ρ(x,y) and θ(x,y) are the optical flow amplitude and optical flow direction of the pixel with coordinates (x,y), respectively.

5. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 4 is characterized in that: In step 3, the method for dividing the lower limb region of the human body is: dividing the lower limb region of the human body into six local regions, representing the right thigh, left thigh, right calf, left calf, right foot and left foot regions respectively; the six local regions are represented as: in They are the right thigh, left thigh, right calf, left calf, right foot and left foot areas.

6. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 5, characterized in that: In step 2, the method for dividing the optical flow field of the lower limb area of ​​the human body is: according to the method for dividing the lower limb area of ​​the human body in step 3, the optical flow field of the lower limb area of ​​the human body is divided into six sub-optical flow fields of local parts; the mathematical expression of the optical flow fields of the six local parts is: where (x,y)∈S t ; Represents the optical flow fields of the right thigh, left thigh, right calf, left calf, right foot and left foot areas respectively.

7. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 6, characterized in that: In step three, the gait motion characteristics are described by the optical flow information of the local part of the lower limbs of the human body, and the local gait motion characteristics are modeled by the local fuzzy optical flow histogram energy map; the calculation method of the local fuzzy optical flow histogram energy map is as follows: (1) Calculate the local optical flow direction histogram: (1.1) Calculate the optical flow direction histogram of each local area of ​​each gait image: in is the optical flow direction histogram of the i-th local area of ​​the t-th frame gait image, i = 1, 2, ..., 6 represents the index of the local part of the human lower limb; B is the number of histogram bins, indicating that the given angle range is evenly divided into B parts; h j is the jth histogram bin, j th bin represents the jth histogram bin; τ is the optical flow amplitude threshold, which means that when the optical flow amplitude ρ(x, y) is greater than or equal to τ, the corresponding optical flow direction θ(x, y) can be effectively counted; (1.2) Concatenate the optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional optical flow direction histogram of the i-th local area: in[·] C Represents the splicing operation in the time dimension, is the two-dimensional optical flow direction histogram of the i-th local area; (1.3) Concatenate the two-dimensional optical flow direction histograms of each local area in the horizontal dimension and calculate the local optical flow direction histogram: H=[H i |i=1,2,…6] W , (19) in[·] W Represents the splicing operation performed in the horizontal dimension, is the local optical flow direction histogram; (2) Calculate the local blurred optical flow direction histogram: (2.1) Define the fuzzy optical flow direction histogram of each local area of ​​each gait image: in is the fuzzy optical flow direction histogram of the i-th local area of ​​the t-th frame gait image; X×Y is the image resolution of the local part; μ jk is the optical flow direction θ of the kth pixel k The membership degree of the jth histogram bin enables flexible allocation of optical flow directions to all associated histogram bins; ρ k Represents the optical flow amplitude of the kth pixel; (2.2) Define the fuzzy membership function: Among them, μ jk is the optical flow direction θ of the kth pixel k The degree of membership with the jth histogram bin; φ j is the center point of the jth histogram bin; φ l is the center point of the lth histogram bin; m is the fuzzy coefficient, which is used to control the similarity between different histogram bins; σ is the variance of the Gaussian term, which is used to control the influence between adjacent histogram bins during the fuzzification process; (2.3) First, based on step (1.1), create an optical flow direction histogram with B′ histogram bins, denoted as a fine histogram, and assign the angles in the range of [-π,π) to the fine histogram; create a normal histogram with B histogram bins, denoted as a coarse histogram, where B′>B; based on the fuzzy membership function described in step (2.2), use the fuzzy c-means clustering algorithm to classify all fine histogram bins into B classes, where each class represents a coarse histogram bin, and obtain the membership of all fine histogram bins to each coarse histogram bin, and express it as a membership matrix: where μ(m;σ) is the fuzzy membership function in step (2.2); M is the membership matrix, where m pq is the membership of the qth fine histogram bin to the pth coarse histogram bin; since the number of coarse histogram bins is the same as the number of histogram bins of the fuzzy optical flow direction histogram described in step (2.1), each class can represent a fuzzy optical flow direction histogram bin, so when p = j and the optical flow direction θ k When it belongs to the qth fine histogram bin, m pq =μ jk ; (2.4) Based on step (1.1), calculate the optical flow direction histogram of each local area of ​​each gait image The number of histogram bins is B′; based on the membership matrix obtained in step (2.3), the fuzzy optical flow direction histogram of each local area of ​​each gait image in step (2.1) is calculated by the following formula: (2.5) Concatenate the fuzzy optical flow direction histogram of the i-th local area in the time dimension and calculate the two-dimensional fuzzy optical flow direction histogram of the i-th local area: in is the two-dimensional fuzzy optical flow direction histogram of the i-th local area; (2.6) The two-dimensional fuzzy optical flow direction histograms of each local area are spliced ​​in the horizontal dimension to calculate the local fuzzy optical flow direction histogram: in is the local blurred optical flow direction histogram; (3) Perform an average operation on the local fuzzy optical flow direction histogram in the spatial dimension and calculate the local fuzzy optical flow histogram energy map: in is the local blurred optical flow histogram energy map.

8. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 1 or 7, characterized in that: In step four, the histogram feature learning network uses the local fuzzy optical flow histogram energy map as the network input, mines the intrinsic gait motion characteristics of the local fuzzy optical flow histogram energy map by training the network, maps the histogram features to a discriminative deep feature space, and outputs a discriminative deep feature vector.

9. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 8, characterized in that: In step 4, the histogram feature learning network includes a nonlinear enhancement module, a feature extraction module and a downsampling module connected in sequence; The nonlinear enhancement module includes two nonlinear enhancement layers connected in sequence, each nonlinear enhancement layer includes a 1×1 convolution layer, a batch normalization layer and a linear rectification function connected in sequence, wherein the 1×1 convolution layer is used to change the number of feature maps to enhance the nonlinear expression ability of features; The feature extraction module includes dense connection module I, dense connection module II, dense connection module III and dense connection module IV, wherein dense connection module I takes the output of the nonlinear enhancement module as input, and any subsequent dense connection module takes the splicing result of the output of all previous dense connection modules and the output of the nonlinear enhancement module in the channel dimension as input; the number of output feature maps of dense connection module I, dense connection module II, dense connection module III and dense connection module IV is k, and k is a pre-set positive integer; dense connection module Ⅰ, dense connection module II, dense connection module III and dense connection module IV respectively include 4, 6, 6, and 8 groups of dense convolutional layers. For a dense connection module, any dense convolutional layer takes the concatenation of the output of all previous dense convolutional layers and the original input of the dense connection module in the channel dimension as input, and outputs k feature maps. The dense convolutional layer includes a batch normalization layer, a linear rectification function, a 1×1 convolutional layer, a batch normalization layer, a linear rectification function and a 3×3 convolutional layer connected in sequence; The downsampling module includes two parallel connected pooling layers, the two pooling layers are a 3×3 adaptive maximum pooling layer and a 3×3 adaptive average pooling layer respectively; the downsampling module takes the output of the feature extraction module as input, performs dimensionality reduction processing on the input, and outputs two sets of feature maps with the same dimensional size; The output deep features of the histogram feature learning network are the concatenation of the output features of the downsampling modules and are expanded into a one-dimensional feature vector.

10. The frontal perspective gait recognition method based on local fuzzy optical flow histogram energy map according to claim 1 or 9, characterized in that: In step four, the gait recognition method calculates the difference between the registered gait representation and the verified gait representation in the deep feature vector in a discriminative deep feature space, returns the identity of the registered gait representation with the smallest distance to the verified gait representation as the identity recognition result, and realizes gait recognition from the frontal perspective.

Images