Event camera-based multi-scale point cloud feature fusion human body posture estimation method

Abstract

The invention relates to the technical field of computer vision, and discloses a multi-scale point cloud feature fusion human body posture estimation method based on an event camera, which comprises the following steps: acquiring event original flow information by using the event camera, converting the event original flow information into a point cloud voxel block, carrying out attribute feature extraction on the point cloud voxel blocks to obtain coordinate attributes and feature attributes; inputting the coordinate attributes and the feature attributes into a constructed graph convolutional neural network to obtain scale features output by each layer of graph convolutional neural network; the self-defined feature tensor and the multi-scale features are sequentially input into a multi-layer cross attention module together to be updated and fused for multiple times, and a result output by the cross attention module of the last layer serves as an updated global feature tensor; processing the global feature tensor to generate a final prediction heat map, and estimating a human body posture by using the prediction heat map; according to the invention, the problems of low precision and poor robustness of an existing human body posture estimation method based on an event camera are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a multi-scale point cloud feature fusion human pose estimation method based on an event camera. BACKGROUND

[0002] With the development of artificial intelligence and robot vision, human pose estimation has a wide range of applications in the fields of motion analysis, human-computer interaction, augmented reality, etc. In the traditional method, it mainly relies on a traditional camera based on a synchronous frame to capture images, but this method has problems such as blurring, large delay, poor adaptability to extreme light environments when capturing motion images. As a new type of visual sensor, the event camera can asynchronously record the changes of pixels with extremely high temporal resolution, and has the advantages of low delay, low power consumption, high dynamic range, etc. However, the event stream data obtained by the event camera is sparse and lacks texture information, resulting in poor feature extraction effect based on traditional convolutional neural networks. Therefore, the existing technology cannot effectively combine the spatiotemporal characteristics of event stream data and graph structure modeling, and there are problems of low accuracy and poor robustness in human pose estimation. SUMMARY

[0003] The present application provides a multi-scale point cloud feature fusion human pose estimation method based on an event camera to solve the problems of low accuracy and poor robustness in existing human pose estimation methods based on an event camera.

[0004] In order to achieve the above purpose, the present application realizes the technical scheme as follows: The present application provides a multi-scale point cloud feature fusion human pose estimation method based on an event camera, comprising: S1, obtaining event original stream information by using an event camera, converting the event original stream information into a point cloud voxel block, and performing attribute feature extraction on the point cloud voxel block to obtain coordinate attributes and feature attributes; S2, inputting the coordinate attributes and the feature attributes into a constructed graph convolutional network to obtain scale features output by each layer of the graph convolutional neural network; S3, inputting a self-defined feature tensor into a cross-attention module together with multi-scale features for multi-scale updating, and taking the output result of the last layer of the cross-attention module as a global feature tensor; S4, processing the global feature tensor to generate a final prediction heat map, and estimating a human pose by using the prediction heat map.

[0005] Optionally, in S1, the method for converting the asynchronous event stream into a point cloud voxel block representation is realized by using an EV-VGCNN model, comprising: dividing the event stream into a spatiotemporal voxel grid; Generating feature attributes based on event density and polarity within a voxel ; Taking voxel space coordinates as coordinate attributes .

[0006] Optionally, in the S2, the graph convolution network is a dynamic aggregation layer structure, and the operation includes: Constructing dynamic graph edge connections according to the spatiotemporal proximity of the point cloud voxel block; Aggregating neighbor node features through a multi-layer perception; Each layer outputs features Corresponding to different scale spatiotemporal receptive fields.

[0007] Optionally, in the S3, the custom feature tensor is sequentially input into the cross-attention module together with the scale features, including: The scale features output by each layer of the graph convolutional neural network are respectively 、 ,... The custom feature tensor and the scale features of the first layer of the graph convolutional neural network are input into the cross-attention module to obtain the output of the cross-attention module, and the output of the cross-attention module is taken as the first updated custom feature tensor; The first updated custom feature tensor and the scale features of the second layer of the graph convolutional neural network are input into the cross-attention module to obtain the output of the cross-attention module, and the output of the cross-attention module is taken as the second updated custom feature tensor; The custom feature tensor is updated in the above manner until the updated custom feature tensor and the scale features of the last layer of the graph convolutional neural network are input into the cross-attention module to obtain the output of the cross-attention module, and the output of the cross-attention module is taken as the global feature tensor.

[0008] Optionally, in the S3, the custom feature tensor is initialized using a normal distribution with a mean of 0 and a standard deviation of 0.2, and the custom feature tensor is sequentially fused with the outputs 、 ,... of the graph convolutional networks at different levels, and the function expression of the feature fusion satisfies the following relationship: ; In the formula, represents a cross-attention module, represents a custom feature tensor input of the first cross-attention module, represents a feature output of the first EDAL module, represents an output of the first cross-attention module; Wherein, the cross-attention module comprises a cross-attention block and a plurality of self-attention blocks; The updated custom feature tensor is dimensionally unified and standardized with the feature output of the first +1 EDAL module, and the function expression of the processing satisfies the following relationship: ; In the formula, is a linear layer operation, used to adjust the dimension of the updated custom feature tensor ; is layer normalization, used to standardize the output of each layer neuron of a single sample.

[0009] Optionally, in the S4, the global feature tensor is processed, comprising: Adjusting the tensor dimension order to align the channel space layout; Reshaping the dimension size to match the target heat map resolution; Mapping the feature to K channels through a linear layer to output a predicted heat map , wherein K is the number of key points.

[0010] Optionally, the method further comprises a model training step, comprising: Training and optimizing the network parameters of the multi-scale graph convolution feature fusion network model by using a back propagation algorithm and a mirror gradient optimization algorithm, and the loss function used to train and optimize the multi-scale graph convolution feature fusion network parameters is: ; ; In the formula, is a heat map converted from the output of the first cross-attention module, is a real label heat map, , , is a hyperparameter, represents the number of key points in the predicted heat map; wherein the hyperparameters can be adjusted according to different task requirements.

[0011] Advantages: The event camera based multi-scale point cloud feature fusion human pose estimation method provided by the application can effectively capture microsecond-level dynamic changes of human motion by converting asynchronous event streams into a space-time graph structure and extracting multi-scale space-time features by using a multi-level graph convolution network.Combining a cross-layer attention fusion mechanism, the application adaptively integrates feature information of different receptive fields, solves the problem of texture loss caused by the sparsity of event stream data, adopts an iterative feature fusion architecture, updates feature tensors layer by layer through a lightweight attention module, avoids redundant feature calculation, represents point cloud voxel blocks to compress event data volume, reduces input information redundancy, and dynamically constructs graph edge connections according to space-time proximity, thereby adapting to the non-rigid deformation of human joints. The application extracts multi-level event point cloud voxel block feature attributes and fuses them to obtain multi-level features and convert them into a heat map for output based on a graph convolutional neural network and a cross-attention mechanism: asynchronous event streams are converted into point cloud voxel blocks and selected to reduce model input information redundancy, and point cloud voxel block coordinate attributes and feature attributes are extracted;the point cloud voxel block attributes are input into a multi-layer graph convolution network to obtain point cloud voxel block attribute features at different levels and save them;the self-defined variables are sequentially input into the cross-attention module with the point cloud voxel block attribute features at different levels to extract and update the features, obtain the final multi-level feature fusion variables, and convert them into a heat map to predict human joint points.The application solves the problems of event camera based event frame deep learning model input data redundancy, large model parameter quantity, and high computational complexity based on a graph convolutional network and a cross-attention mechanism, can effectively extract event stream features for human pose estimation, reduce model training cost, speed up model inference, and can be widely applied to devices with small size, low storage capacity, and insufficient computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0012] Fig. 1 The flowchart of the event camera based multi-scale point cloud feature fusion human pose estimation method of the preferred embodiment of the application is shown in the figure. Fig. 2 The model structure diagram of the preferred embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0013] The technical solutions of the application will be described clearly and completely below.It should be apparent that the described embodiments are only part of the embodiments of the application, not all the embodiments.Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0014] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the meanings as understood by a person of ordinary skill in the art to which the present application pertains. The terms "first", "second", and similar terms are used herein merely to distinguish one element from another, and are not intended to imply any order or sequence, or any importance. Similarly, the terms "one", "another", and similar terms are not limited to one or more, but can include one or more. The terms "connected", "coupled", or similar terms are not limited to a direct connection or coupling, but can include an indirect connection or coupling, unless otherwise defined. The terms "upper", "lower", "left", "right", and similar terms are used merely to indicate relative positional relationships, and can be changed when the absolute positions of the described objects are changed.

[0015] Please refer to Figs. 1-2 The embodiment of the present application provides a multi-scale point cloud feature fusion human pose estimation method based on an event camera, comprising: S1, acquiring event original stream information by using an event camera, converting the event original stream information into a point cloud voxel block, and performing attribute feature extraction on the point cloud voxel block to obtain coordinate attributes and feature attributes; S2, inputting the coordinate attributes and the feature attributes into a constructed graph convolution network to obtain scale features output by each layer of the graph convolution neural network; S3, sequentially inputting a self-defined feature tensor into a cross-attention module together with multi-scale features for multi-scale updating, and taking an output result of a last layer of the cross-attention module as a global feature tensor; S4, processing the global feature tensor to generate a final prediction heat map, and estimating a human pose by using the prediction heat map.

[0016] Optionally, in S1, the method for converting the asynchronous event stream into a point cloud voxel block representation is implemented by using an EV-VGCNN model, comprising: dividing the event stream into a spatiotemporal voxel grid; generating feature attributes based on event density and polarity in the voxel ; taking voxel space coordinates as coordinate attributes .

[0017] In the above embodiment, the method for converting the asynchronous event stream into a point cloud voxel block representation and extracting coordinate attributes and feature attributes is implemented by using the method in the EV-VGCNN.

[0018] Optionally, in S2, the graph convolution network is a dynamic aggregation layer structure, and the operation thereof comprises: constructing dynamic graph edge connections according to spatiotemporal proximity of the point cloud voxel block Features of neighboring nodes are aggregated using a multilayer perceptron. Output features of each layer Corresponding to different scales of spatiotemporal receptive fields.

[0019] In the above embodiments, the graph convolutional network module is implemented using a dynamic aggregation layer (EDAL) structure.

[0020] Optionally, in step S3, the custom feature tensor is input into the cross-attention module along with the scale feature, including: The scale features output by each layer of the graph convolutional neural network are as follows: , ... Custom feature tensor Scale features of the first layer of the graph convolutional neural network The input is fed into the cross-attention module to obtain the output of the cross-attention module. and the output of the cross-attention module As the custom feature tensor after the first update; The custom feature tensor after the first update Scale features of the second-layer graph convolutional neural network The input is fed into the cross-attention module to obtain the output of the cross-attention module. and the output of the cross-attention module As a custom feature tensor after the second update; Update the custom feature tensor using the above method until the updated custom feature tensor matches the scale features of the last layer of the graph convolutional neural network. The input is fed into the cross-attention module to obtain the output of the cross-attention module. The output of the cross-attention module As a global feature tensor.

[0021] Optionally, in S3, a custom feature tensor is defined. Initialize using a normal distribution with a mean of 0 and a standard deviation of 0.2, and define a custom feature tensor. The outputs are sequentially fed into graph convolutional networks of different levels. , ... The function expression for feature fusion satisfies the following relation: ; In the formula, This indicates a cross-attention module. Indicates the first Custom feature tensors are input to each cross-attention module. representing the feature output of the EDAL module, representing the output of the cross-attention module, wherein the cross-attention module comprises a cross-attention block and a plurality of self-attention blocks; updating the custom feature tensor with the feature output of the EDAL module, wherein, is a linear layer operation, used to adjust the dimension of the updated custom feature tensor is layer normalization, used to standardize the output of each layer neuron of a single sample.

[0022] Optionally, in the S4, the global feature tensor is processed, including: adjusting the tensor dimension order to align the channel space layout; resizing the dimension to match the target heat map resolution; mapping the features to K channels through a linear layer to output a predicted heat map wherein K is the number of key points.

[0023] Optionally, the method further comprises a model training step, including: training and optimizing the network parameters of the multi-scale graph convolution feature fusion network model by using a back propagation algorithm and a mirror gradient optimization algorithm, and the loss function used to train and optimize the multi-scale graph convolution feature fusion network parameters is: wherein, is the heat map converted from the output of the cross-attention module, is the real label heat map, is a hyperparameter, representing the number of key points in the predicted heat map; wherein the hyperparameter can be adjusted according to different task requirements.

[0024] ​​​​​​​​​​The preferred embodiments of the present application have been described above in detail. It should be understood that modifications and variations to the present application can be made by those skilled in the art without departing from the spirit and scope of the application. Accordingly, it should be understood that the application is not limited to the preferred embodiments described above, but is intended to cover all modifications and variations within the scope and spirit of the application.

Claims

1. A method for human pose estimation based on multi-scale point cloud feature fusion using event cameras, characterized in that, include: S1. Use the event camera to acquire raw event stream information, convert the raw event stream information into point cloud voxel blocks, and extract the attribute features of the point cloud voxel blocks to obtain coordinate attributes and feature attributes. S2. Input the coordinate attributes and feature attributes into the constructed graph convolutional network to obtain the scale features output by each layer of the graph convolutional neural network; S3. Input the custom feature tensor along with the multi-scale features into the cross-attention module for multi-scale updates, and use the output of the last layer of the cross-attention module as the global feature tensor. S4. Process the global feature tensor to generate the final prediction heatmap, and use the prediction heatmap to estimate the human pose.

2. The human pose estimation method based on multi-scale point cloud feature fusion using an event camera according to claim 1, characterized in that, In step S1, the method for converting the raw event stream information into point cloud voxel blocks is implemented using the EV-VGCNN model, including: The raw event stream information is segmented into a spatiotemporal voxel grid; Based on event density and polarity generation characteristics within the spatiotemporal voxel grid ; Using voxel space coordinates as coordinate attributes The event's raw flow information is converted into point cloud voxel blocks using feature attributes and coordinate attributes.

3. The human pose estimation method based on multi-scale point cloud feature fusion using an event camera according to claim 1, characterized in that, In step S2, the graph convolutional network is a dynamic aggregation layer structure, and its operations include: Dynamic graph edge connections are constructed based on the spatiotemporal proximity of point cloud voxel blocks; Features of neighboring nodes are aggregated using a multilayer perceptron. Output features of each layer Corresponding to different scales of spatiotemporal receptive fields.

4. The human pose estimation method based on multi-scale point cloud feature fusion using an event camera according to claim 1, characterized in that, In step S3, the custom feature tensor is sequentially input into the cross-attention module along with the scale feature, including: The scale features output by each layer of the graph convolutional neural network are as follows: , ... Custom feature tensor Scale features of the first layer of the graph convolutional neural network The input is fed into the cross-attention module to obtain the output of the cross-attention module. and the output of the cross-attention module As the custom feature tensor after the first update; The custom feature tensor after the first update Scale features of the second-layer graph convolutional neural network The input is fed into the cross-attention module to obtain the output of the cross-attention module. and the output of the cross-attention module As a custom feature tensor after the second update; Update the custom feature tensor using the above method until the updated custom feature tensor matches the scale features of the last layer of the graph convolutional neural network. The input is fed into the cross-attention module to obtain the output of the cross-attention module. The output of the cross-attention module As a global feature tensor.

5. The multi-scale point cloud feature fusion human pose estimation method based on event camera according to claim 3, characterized in that, In S3, the custom feature tensor Initialize using a normal distribution with a mean of 0 and a standard deviation of 0.2, and define a custom feature tensor. The outputs are sequentially fed into graph convolutional networks of different levels. , ... The function expression for feature fusion satisfies the following relation: ； In the formula, This indicates a cross-attention module. Indicates the first Custom feature tensors are input to each cross-attention module. Indicates the first The feature outputs of each EDAL module Indicates the first The output of each cross-attention module; The cross-attention module includes one cross-attention block and multiple self-attention blocks; Update the custom feature tensor Same as the first The feature outputs of +1 EDAL module undergo dimensionality unification and standardization processing, and the processing function expression satisfies the following relationship: ； In the formula, This is a linear layer operation used to adjust the updated custom feature tensor. Dimension Layer normalization is used to standardize the output of each layer of neurons in a single sample.

6. The human pose estimation method based on multi-scale point cloud feature fusion using an event camera according to claim 1, characterized in that, In step S4, the global feature tensor is processed, including: Adjust the tensor dimension order to align the channel space layout; Resize dimensions to match the target heatmap resolution; The features are mapped to K channels through a linear layer, and a predicted heatmap is output. , where K is the number of key points.

7. The human pose estimation method based on multi-scale point cloud feature fusion using an event camera according to claim 1, characterized in that, The method further includes a model training step, which includes: The network parameters of the multi-scale graph convolutional feature fusion network model are trained and optimized using the backpropagation algorithm and the mirror gradient optimization algorithm. The loss function used for training and optimizing the multi-scale graph convolutional feature fusion network parameters is: ； ； In the formula, For the first The output of each cross-attention module is converted into a heatmap. This is a real-world heatmap with tags. , , For hyperparameters, L represents the number of key points in the predicted heatmap, and L is the loss function. The hyperparameters can be adjusted according to different task requirements.

Images