Posture estimation method and system based on multi-modal fusion, electronic equipment and medium

Abstract

The invention discloses a posture estimation method and system based on multi-modal fusion, electronic equipment and a medium, and the method comprises the steps: obtaining a to-be-detected image, a depth image and millimeter wave signal data, and enabling the to-be-detected image to comprise at least one target object; performing feature projection on the to-be-detected image and the depth image through a preset deep neural network to obtain first feature data, and performing feature projection on the millimeter wave signal data to obtain second feature data; inputting the first feature data and the second feature data into an encoder module for feature fusion, and outputting aggregated feature data; performing feature separation on the aggregated feature data through a decoder module to obtain a category vector and a key point vector; and estimating the target attitude of the target object according to the category vector and the key point vector. In the embodiment of the invention, the accuracy of human body posture estimation is improved by processing the data of different modalities.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a posture estimation method, system and electronic equipment based on multimodal fusion. Background Art

[0002] Human pose estimation is a computer vision technology that aims to identify and track key parts of the human body from various types of data sources. This includes the positions of joints such as the head, arms, wrists, waist, legs, etc., usually represented as key points. This technology is of great significance for understanding human movements and behaviors, and is widely used in video surveillance, interactive games, motion analysis, virtual reality and other fields.

[0003] Although human pose estimation technology has shown great application potential in many fields, it still faces some challenges and shortcomings in practical applications. Traditional human pose estimation methods are mainly based on single-modal data. This method lacks certain flexibility and cannot fully utilize the richness of data to improve model performance in multimodal scenarios. This limits the performance of the model in complex scenarios and further reduces the accuracy of human pose estimation. Summary of the invention

[0004] The embodiments of the present application provide a posture estimation method, system, electronic device and storage medium based on multimodal fusion, which improve the accuracy of human posture estimation by processing data of different modalities.

[0005] In a first aspect, an embodiment of the present application provides a posture estimation method based on multimodal fusion, the method comprising:

[0006] Acquire an image to be detected, a depth image, and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and an image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0007] Performing feature projection on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and performing feature projection on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module;

[0008] Inputting the first feature data and the second feature data into the encoder module for feature fusion, and outputting aggregated feature data;

[0009] Performing feature separation on the aggregated feature data by the decoder module to obtain a category vector and a key point vector;

[0010] The target posture of the target object is estimated according to the category vector and the key point vector.

[0011] In some embodiments, the deep neural network includes a first feature projection module and a second feature projection module; the feature projection of the image to be detected and the depth image by a preset deep neural network to obtain the first feature data, and the feature projection of the millimeter wave signal data to obtain the second feature data includes:

[0012] Inputting the image to be detected and the depth image into the first feature projection module, so that the first feature projection module performs a convolution operation on the image to be detected and the depth image, and outputs first feature data;

[0013] The millimeter wave signal data is input into the second feature projection module, so that the second feature projection module performs a convolution operation on the millimeter wave signal data and outputs second feature data.

[0014] In some embodiments, the encoder module includes a first encoder module and a second encoder module; the step of inputting the first feature data and the second feature data into the encoder module for feature fusion and outputting aggregated feature data includes:

[0015] Inputting the first feature data and the second feature data into the first encoder module, so that the first encoder module merges the first feature data and the second feature data to obtain a merged feature map;

[0016] Performing feature aggregation on the merged feature map by the first encoder module to obtain a feature aggregation map;

[0017] The feature aggregation graph is input into the second encoder module, so that the second encoder module performs dimensional feature weighting on the feature aggregation graph and outputs aggregated feature data.

[0018] In some embodiments, the decoder module includes a first fully connected layer and a second fully connected layer; the step of performing feature separation on the aggregated feature data through the decoder module to obtain a category vector and a key point vector includes:

[0019] Separating the aggregated features row by row through the decoder module to obtain intermediate features;

[0020] Inputting the intermediate features into the first fully connected layer for dimensionality reduction processing, and outputting the reduced dimensionality features;

[0021] Input the reduced-dimensionality feature into the second fully connected layer for dimensionality increase processing, and output the target feature;

[0022] The decoder module performs a full connection operation on the target feature to generate a category vector and a key point vector.

[0023] In some embodiments, the method further comprises:

[0024] The deep neural network is trained for key point prediction using a preset point-to-point mean square error loss function.

[0025] In some embodiments, performing feature aggregation on the merged feature map by the first encoder module to obtain a feature aggregation map includes:

[0026] Performing an averaging operation on the row features of the merged feature map through the first encoder module to obtain a first weight vector;

[0027] The features of the merged feature map are weighted in the column dimension according to the first weight vector to obtain a feature aggregation map.

[0028] In some embodiments, inputting the feature aggregation graph into the second encoder module so that the second encoder module performs dimensional feature weighting on the feature aggregation graph and outputs aggregate feature data includes:

[0029] Performing a transposition operation on the feature aggregation graph through the second encoder module to flip the row features and column features of the feature aggregation graph to obtain flipped intermediate features;

[0030] Performing feature aggregation on the flipped intermediate features through the second encoder module to obtain a second weight vector;

[0031] The flipped intermediate features are weighted in row dimension according to the second weight vector to obtain aggregated feature data.

[0032] In a second aspect, an embodiment of the present application further provides a posture estimation system based on multimodal fusion, the system comprising:

[0033] A data acquisition module, used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0034] A feature projection module, used to perform feature projection on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and perform feature projection on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module;

[0035] a feature fusion module, configured to input the first feature data and the second feature data into the encoder module for feature fusion, and output aggregated feature data;

[0036] A feature separation module, used for performing feature separation on the aggregated feature data through the decoder module to obtain a category vector and a key point vector;

[0037] A posture estimation module is used to estimate the target posture of the target object according to the category vector and the key point vector.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the posture estimation method based on multimodal fusion as described in the first aspect is implemented.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the posture estimation method based on multimodal fusion as described in the first aspect.

[0040] The posture estimation method based on multimodal fusion provided in the embodiment of the present application has at least the following beneficial effects: first, the image to be detected, the depth image and the millimeter wave signal data are obtained, wherein the image to be detected includes at least one target object, and the geometric shape of the visible surface of the target object can be directly obtained by obtaining the depth image, and the distance between the target object and the image acquisition device can be determined, and the target object can be accurately detected and positioned by obtaining the millimeter wave signal data, and the subsequent detection of the target object is facilitated by obtaining different types of data. The image to be detected and the depth image are feature projected by a preset deep neural network to obtain the first feature data, and the millimeter wave signal data is feature projected to obtain the second feature data, so as to realize feature mapping of different types of data, facilitate subsequent feature fusion, and enhance the feature expression ability of the first feature data and the second feature data. Afterwards, the first feature data and the second feature data are input into the encoder module for feature fusion to improve the feature expression ability of the aggregated feature data, and the aggregated feature data is output, which can improve the scale invariance of the feature. Through adaptive feature fusion, the deep neural network can better process multimodal data and improve the generalization ability of the deep neural network in different environments and conditions. The aggregated feature data is then separated by the decoder module to improve the richness of the data. The features aggregated in the encoder are effectively separated, and feature information at different levels can be retained, enhancing the model's ability to capture details, and obtaining category vectors and key point vectors, which is convenient for subsequent judgment of the posture category through category vectors and key point vectors, thereby improving the accuracy of human posture estimation. Finally, the target posture of the target object is estimated according to the category vector and key point vector to achieve accurate estimation of the posture.

[0041] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the examples of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0043] Figure 1 is a flowchart of a specific method of a posture estimation method based on multimodal fusion provided in an embodiment of the present application;

[0044] Figure 2 It is a specific flow chart of performing feature projection on the image to be detected and the depth image through a preset deep neural network provided in an embodiment of the present application;

[0045] Figure 3 is a specific flow chart of inputting the first feature data and the second feature data into the encoder module for feature fusion provided by an embodiment of the present application;

[0046] Figure 4 It is a specific flow chart of feature separation of aggregated feature data through a decoder module provided in an embodiment of the present application;

[0047] Figure 5 is a specific flow chart of a posture estimation method based on multimodal fusion provided by another embodiment of the present application;

[0048] Figure 6 It is a specific flow chart of performing feature aggregation on a merged feature map by a first encoder module provided in an embodiment of the present application;

[0049] Figure 7 It is a specific flow chart of the second encoder module provided in the embodiment of the present application performing dimensional feature weighting on the feature aggregation graph;

[0050] Figure 8 is a structural diagram of a posture estimation system based on multimodal fusion provided in an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application;

[0052] Fig.10 is a module schematic diagram of a first feature projection module provided in an example of the present application;

[0053] Fig.11 is a module schematic diagram of a second feature projection module provided in an example of the present application;

[0054] Fig.12 is a module schematic diagram of an encoder module provided in an example of the present application;

[0055] Fig.13 is a module schematic diagram of a decoder module provided in an example of the present application;

[0056] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] The posture estimation method based on multimodal fusion provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0061] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] Human pose estimation is a computer vision technology that aims to identify and track key parts of the human body from various types of data sources. This includes the positions of joints such as the head, arms, wrists, waist, legs, etc., usually represented as key points. This technology is of great significance for understanding human movements and behaviors, and is widely used in video surveillance, interactive games, motion analysis, virtual reality and other fields.

[0063] Although human pose estimation technology has shown great application potential in many fields, it still faces some challenges and shortcomings in practical applications. Traditional human pose estimation methods are mainly based on single-modal data. This method lacks certain flexibility and cannot fully utilize the richness of data to improve model performance in multimodal scenarios. This limits the performance of the model in complex scenarios and further reduces the accuracy of human pose estimation.

[0064] In order to solve the above problems, the embodiments of the present application provide a posture estimation method, system, electronic device and storage medium based on multimodal fusion. First, the image to be detected, the depth image and the millimeter wave signal data are obtained, wherein the image to be detected includes at least one target object. By obtaining the depth image, the geometric shape of the visible surface of the target object can be directly obtained, and the distance between the target object and the image acquisition device can be determined. By obtaining the millimeter wave signal data, the target object can be accurately detected and positioned. By obtaining different types of data, the subsequent detection of the target object is facilitated. The image to be detected and the depth image are feature projected by a preset deep neural network to obtain the first feature data, and the millimeter wave signal data is feature projected to obtain the second feature data, so as to realize feature mapping of different types of data, facilitate subsequent feature fusion, and enhance the feature expression ability of the first feature data and the second feature data. Afterwards, the first feature data and the second feature data are input into the encoder module for feature fusion to improve the feature expression ability of the aggregated feature data, and the aggregated feature data is output, which can improve the scale invariance of the feature. Through adaptive feature fusion, the deep neural network can better process multimodal data and improve the generalization ability of the deep neural network in different environments and conditions. The aggregated feature data is then separated by the decoder module to improve the richness of the data. The features aggregated in the encoder are effectively separated, and feature information at different levels can be retained, enhancing the model's ability to capture details, and obtaining category vectors and key point vectors, which is convenient for subsequent judgment of the posture category through category vectors and key point vectors, thereby improving the accuracy of human posture estimation. Finally, the target posture of the target object is estimated according to the category vector and key point vector to achieve accurate estimation of the posture.

[0065] Reference Figure 1 , Figure 11 is a flowchart of a specific method of a posture estimation method based on multimodal fusion provided in an embodiment of the present application. In some embodiments, the posture estimation method based on multimodal fusion includes but is not limited to steps S101 to S105.

[0066] Step S101, obtaining an image to be detected, a depth image and millimeter wave signal data.

[0067] It should be noted that the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by the sensor.

[0068] In step S101 of some embodiments, an image to be detected, a depth image and millimeter wave signal data are obtained, wherein the image to be detected is obtained by performing image acquisition by an image acquisition device, and the depth image can be obtained by a depth sensor, laser scanning technology, etc., and is used to reflect the distance between the target object and the image acquisition device. Specifically, the depth image uses the distance (depth) value of each point in the scene acquired by the image acquirer as the pixel value of the image, and the millimeter wave signal data is obtained by a sensor using a high-frequency millimeter wave signal for ranging, which facilitates the subsequent detection of the target object through different images and data, and can simultaneously support the detection of multiple data types, facilitating the subsequent multi-modal target detection.

[0069] It should be noted that the embodiment of the present application is described using a millimeter-wave radar as an example. The millimeter-wave radar transmits a series of frequency-modulated signals, which are reflected back to the radar after contacting environmental objects. The radar receives the reflected signals through the receiving antenna and measures the signal's time delay, frequency, amplitude and other parameters, and processes the received millimeter-wave signal data.

[0070] It can be understood that the target detection method based on the multimodal model in the embodiment of the present application can be applied to public safety monitoring, crowd behavior analysis, interactive media and advanced driver assistance systems, etc., wherein the target object can be a vehicle, a human body, etc., and the embodiment of the present application does not make specific restrictions.

[0071] Step S102, feature projection is performed on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and feature projection is performed on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module.

[0072] In step S102 of some embodiments, feature projection is performed on the image to be detected and the depth image through a preset deep neural network, reducing the spatial resolution of the image to be detected and the depth image, further reducing the amount of data that the image processing algorithm needs to process, thereby reducing the computational complexity and computational time, obtaining first feature data, and realizing feature detection of the image to be detected and the depth image. Feature projection is performed on the millimeter wave signal data, increasing the feature point dimension and feature length of the millimeter wave signal data, obtaining second feature data, and realizing feature extraction of the millimeter wave signal data.

[0073] Step S103: input the first feature data and the second feature data into an encoder module for feature fusion, and output aggregated feature data.

[0074] In step S103 of some embodiments, the first feature data and the second feature data are input into the encoder module for feature fusion, providing more accurate position and boundary information, enhancing features such as edge, texture and shape of the target, thereby improving accuracy, and outputting aggregated feature data, thereby improving the performance of target detection and recognition tasks and making the model more robust and accurate.

[0075] Step S104, performing feature separation on the aggregated feature data through a decoder module to obtain a category vector and a key point vector.

[0076] In step S104 of some embodiments, the aggregated feature data is feature separated by a decoder module to obtain a category vector and a key point vector, wherein the category vector is used to describe the probability values ​​of human targets and non-human targets, and the key point vector is used to characterize the key point coordinates of the human body, so as to facilitate the subsequent accurate estimation of the posture of the target object.

[0077] Step S105 , estimating the target posture of the target object according to the category vector and the key point vector.

[0078] In step S105 of some embodiments, the coordinates of the target posture of the target object are determined by the key point vector, and the category of the target posture is marked by the category vector, so as to estimate the posture of the target object and improve the accuracy of posture estimation.

[0079] Reference Figure 2 , Figure 2 It is a specific flow chart of performing feature projection on the image to be detected and the depth image through a preset deep neural network provided in an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S201 to S202.

[0080] It should be noted that the deep neural network includes a first feature projection module and a second feature projection module, wherein the first feature projection module is used to process image data, and the second feature projection module is used to process millimeter wave data.

[0081] It is worth noting that the first feature projection module in the embodiment of the present application includes one 1*1 convolutional layer and five 3*3 convolutional layers, and the second feature projection module includes two 1*1 convolutional layers.

[0082] Step S201: input the image to be detected and the depth image into a first feature projection module, so that the first feature projection module performs a convolution operation on the image to be detected and the depth image, and outputs first feature data.

[0083] In step S201 of some embodiments, in the process of performing feature projection on the image to be detected and the depth image through a preset deep neural network, the image to be detected and the depth image are input into the first feature projection module, so that the first feature projection module performs convolution operation on the image to be detected and the depth image. Specifically, the image to be detected and the depth image first pass through a 1*1 convolution layer, and the 1*1 convolution layer performs 1*1 convolution operation, batch standardization operation and Relu activation operation on the image to be detected and the depth image, at which time the number of channels will be increased, and then pass through five 3*3 convolution layers, and the 3*3 convolution layer performs 3*3 convolution operation, batch standardization operation and Relu activation operation on the image to be detected and the depth image, thereby reducing the spatial resolution of the image to be detected and the depth image, and further reducing the amount of data that needs to be processed by the image processing algorithm, thereby reducing the computational complexity and computational time. Finally, the first feature projection module performs a matrix conversion operation on the features processed by the 3*3 convolution layer, outputs the first feature data, and realizes feature detection of the image to be detected and the depth image.

[0084] Step S202: input the millimeter wave signal data into the second feature projection module, so that the second feature projection module performs a convolution operation on the millimeter wave signal data and outputs second feature data.

[0085] In step S202 of some embodiments, the millimeter wave signal data is input into the second feature projection module so that the second feature projection module performs a convolution operation on the millimeter wave signal data. Specifically, the millimeter wave signal data undergoes two 1*1 convolution layers and two matrix transposition operations to increase the feature point dimension and feature length of the millimeter wave signal data, and outputs the second feature data to achieve the extraction of features of the millimeter wave signal data.

[0086] Reference Figure 3 , Figure 3It is a specific flow chart of inputting the first feature data and the second feature data into the encoder module for feature fusion provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to steps S301 to S303.

[0087] It should be noted that the encoder module includes a first encoder module and a second encoder module, wherein an input end of the second encoder module is connected to an output end of the first encoder module.

[0088] Step S301: input the first feature data and the second feature data into a first encoder module, so that the first encoder module merges the first feature data and the second feature data to obtain a merged feature map.

[0089] Step S302: Perform feature aggregation on the merged feature map through the first encoder module to obtain a feature aggregation map.

[0090] Step S303: input the feature aggregation graph into the second encoder module, so that the second encoder module performs dimensional feature weighting on the feature aggregation graph and outputs aggregated feature data.

[0091] In steps S301 to S303 of some embodiments, in the process of inputting the first feature data and the second feature data into the encoder module for feature fusion, the embodiment of the present application first inputs the first feature data and the second feature data into the first encoder module so that the first encoder module performs data merging on the first feature data and the second feature data. By performing data merging, more discriminative and expressive features can be obtained, the prediction performance of the model can be improved, and a merged feature map can be obtained. Then, the first encoder performs feature aggregation on the merged feature map to improve the feature expression ability, so that the model can pay more attention to important features and obtain a feature aggregation map, thereby providing more accurate position and boundary information, enhancing the edge, texture, shape and other features of the target, thereby improving the accuracy. Afterwards, the feature aggregation map is input into the second encoder module so that the second encoder module performs dimensional feature weighting on the feature aggregation map and outputs aggregated feature data, thereby improving the performance of target detection and recognition tasks and making the model more robust and accurate.

[0092] Reference Figure 4 , Figure 4 It is a specific flow chart of feature separation of aggregated feature data by a decoder module provided in an embodiment of the present application. In some embodiments, step S103 specifically includes but is not limited to steps S401 to S404.

[0093] It should be noted that the decoder module includes a first fully connected layer and a second fully connected layer.

[0094] Step S401, the aggregated features are separated row by row through a decoder module to obtain intermediate features.

[0095] Step S402: input the intermediate features into the first fully connected layer for dimensionality reduction processing, and output the reduced dimensionality features.

[0096] Step S403: input the reduced-dimensionality features into the second fully connected layer for dimensionality increase processing, and output the target features.

[0097] Step S404: Perform a full connection operation on the target features through the decoder module to generate a category vector and a key point vector.

[0098] In steps S401 to S404 of some embodiments, in the process of feature separation of aggregated feature data through the decoder module, the embodiments of the present application first separate the aggregated features row by row through the decoder module, thereby enhancing the model's ability to locate details and enhance the ability to express different features, and obtain intermediate features. The intermediate features are then input into the first fully connected layer for dimensionality reduction processing, and the reduced dimensionality features are output. The reduced dimensionality features are input into the second fully connected layer for dimensionality increase processing, and the target features are output. Thereafter, the target features are fully connected through the decoder module to generate category vectors and key point vectors, wherein the category vectors are used to describe the probability values ​​of human targets and non-human targets, and the key point vectors are used to characterize the key point coordinates of the human body, so as to facilitate the subsequent accurate estimation of the posture of the target object.

[0099] Reference Figure 5 , Figure 5 It is a specific flow chart of a posture estimation method based on multimodal fusion provided by another embodiment of the present application. In some embodiments, the method includes but is not limited to step S501.

[0100] Step S501: Perform key point prediction training on the deep neural network using a preset point-to-point mean square error loss function.

[0101] In step S501 of some embodiments, the embodiments of the present application also perform key point prediction training on the deep neural network through a preset point-to-point mean square error loss function, thereby improving the accuracy of the deep neural network in posture estimation.

[0102] Specifically, the point-to-point mean square error loss function is expressed as follows:

[0103]

[0104] Among them, P i and They represent the spatial coordinate vector (x, y, z) of the i-th key point in the real target and the predicted target respectively, and n represents the number of key points (set to 17).

[0105] It is worth noting that each predicted target and the true target are optimally matched according to the mean square error between the predicted key points and the true key points, so as to realize the accurate multi-target human posture estimation in the model during reasoning. Among them, in each training, the embodiment of the present application will calculate the mean square error between the predicted key points and the true key points between each predicted target and the true target, and then sort them from small to large, and each time select the predicted target and the true target with the minimum mean square error for pairing, and then use the loss function to optimize the network model end-to-end (update the parameters of each layer).

[0106] In some embodiments, the embodiments of the present application also train the decoder module through a cross entropy loss function to improve the accuracy of the output vector of the decoder module.

[0107] Specifically, the cross entropy loss function is expressed as follows:

[0108]

[0109] Among them, y i and represent the true label and predicted probability respectively.

[0110] Reference Figure 6 , Figure 6 It is a specific flow chart of performing feature aggregation on the merged feature map by the first encoder module provided in an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S601 to S602.

[0111] It can be understood that the first encoder module in the embodiment of the present application includes two 1*1 convolutional layers.

[0112] Step S601: Perform an averaging operation on the row features of the merged feature map through a first encoder module to obtain a first weight vector.

[0113] Step S602: weighting the features of the merged feature map in the column dimension according to the first weight vector to obtain a feature aggregation map.

[0114] In steps S601 to S602 of some embodiments, in the process of performing feature aggregation on the merged feature map through the first encoder module, the embodiments of the present application first perform an averaging operation on the row features of the merged feature map through the first encoder module. Through the row averaging operation, the dimension of each feature map can be reduced from two dimensions to one dimension, thereby reducing the number of parameters and computational complexity in subsequent processing steps. In addition, the row averaging operation can capture the global information in the feature map to obtain a first weight vector. The features of the merged feature map in the column dimension are then weighted according to the first weight vector to improve the feature expression capability, so that the model can pay more attention to important features to obtain a feature aggregation map. Feature weighting can effectively integrate information between different features, provide more accurate position and boundary information, enhance features such as the edge, texture and shape of the target, and thus improve accuracy.

[0115] Reference Figure 7 , Figure 7 It is a specific flow chart of the second encoder module providing in an embodiment of the present application performing dimensional feature weighting on the feature aggregation graph. In some embodiments, the method includes but is not limited to steps S701 to S703.

[0116] It can be understood that the second encoder module in the embodiment of the present application includes two 1*1 convolutional layers.

[0117] Step S701: Perform a transposition operation on the feature aggregation graph through a second encoder module to flip the row features and column features of the feature aggregation graph to obtain flipped intermediate features.

[0118] Step S702: Perform feature aggregation on the flipped intermediate features through a second encoder module to obtain a second weight vector.

[0119] Step S703: weighting the features of the flipped intermediate features in the row dimension according to the second weight vector to obtain aggregated feature data.

[0120] In steps S701 to S703 of some embodiments, during the process of the second encoder module performing dimensional feature weighting on the feature aggregation graph, the second encoder module performs a transposition operation on the feature aggregation graph to flip the row features and column features of the feature aggregation graph, so that the model can generate a high-resolution image from low-resolution features, and obtain flipped intermediate features, thereby being able to restore the spatial resolution of the feature graph, and then perform feature aggregation on the flipped intermediate features through the second encoder module. Specifically, the embodiment of the present application obtains a second weight vector through feature aggregation operation and two 1*1 convolutional layers, and then performs row-dimensional feature weighting on the flipped intermediate features according to the second weight vector, thereby improving the feature expression capability, so that the model can pay more attention to important features, and obtain aggregated feature data, thereby improving the performance of target detection and recognition tasks, and making the model more robust and accurate.

[0121] See also Figure 8 The embodiment of the present application further provides a posture estimation system based on multimodal fusion, which can implement the above-mentioned posture estimation method based on multimodal fusion, and the system includes:

[0122] The data acquisition module 801 is used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor;

[0123] A feature projection module 802 is used to perform feature projection on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and to perform feature projection on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module;

[0124] A feature fusion module 803, used for inputting the first feature data and the second feature data into the encoder module for feature fusion, and outputting aggregated feature data;

[0125] A feature separation module 804 is used to perform feature separation on the aggregated feature data through a decoder module to obtain a category vector and a key point vector;

[0126] The posture estimation module 805 is used to estimate the target posture of the target object according to the category vector and the key point vector.

[0127] In order to more clearly explain the posture estimation method, system and electronic device based on multimodal fusion of the embodiments of the present application, specific examples are given below for illustration.

[0128] Example 1:

[0129] See also Fig. 9 , Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application.

[0130] In some embodiments, the posture estimation method based on multimodal fusion is modeled as an end-to-end deep neural network. By inputting RGB images, depth images and millimeter wave signal data, it can simultaneously output (predict) human skeleton key points in multiple 3D spaces to achieve human posture estimation. The method mainly includes three key modules, namely, a feature projection module, an encoder module based on global attention enhancement and adaptive feature fusion mechanism, and a decoder module of 3D human key points based on optimal matching loss.

[0131] The following is a detailed description of the posture estimation method based on multimodal fusion in this example in combination with different modules.

[0132] See also Fig.10 , Fig.10 This is a module schematic diagram of a first feature projection module provided in an example of the present application.

[0133] In some embodiments, according to different modalities of input data, the feature projection module is mainly divided into two types: a feature projection module for image data (a first feature projection module) and a feature projection module for millimeter wave data (a second feature projection module).

[0134] The first feature projection module is mainly responsible for processing image data, including RGB images (dimension: 800*640*3) and depth images (dimension: 800*640*1). The image data first passes through a 1*1 convolution layer (including 1*1 convolution operation, batch normalization operation and Relu activation operation) to increase the number of channels (256), and then passes through 5 3*3 convolution layers (including 3*3 convolution operation, batch normalization operation and Relu activation operation) to reduce the spatial resolution (25*20), and finally outputs a 500*256 projection feature map through matrix conversion operation.

[0135] See also Fig.11 , Fig.11 It is a module schematic diagram of the second feature projection module provided in an example of the present application.

[0136] In some embodiments, the second feature projection module is mainly responsible for processing millimeter wave data (dimension: 128*5), where 128 represents the number of point clouds (feature points) and 5 represents the feature length. Each feature point includes spatial coordinates (x, y, z), signal transmission rate and intensity characteristics. The millimeter wave data is processed through two 1*1 convolutional layers and two matrix transposition operations, which increase the feature point dimension and feature length respectively, and output a 500*256 feature map.

[0137] See also Fig.12 , Fig.12 This is a module diagram of an encoder module provided in an example of the present application.

[0138] In some embodiments, the encoder module is responsible for processing all feature maps from the projection module. In the first stage, these feature maps are first merged into a 1500*256 feature map, and then a 1*256 weight vector is generated through a feature aggregation operation (averaging the row features) and two 1*1 convolutional layers. The weight vector can be used to weight the column-dimensional features of the input feature map to improve the feature expression capability. In the second stage, the weighted feature map flips the row and column features through a transposition operation, and then a 1*1500 weight vector is generated through a feature aggregation operation and two 1*1 convolutional layers. The weight vector can be used to weight the row-dimensional features of the input feature map. Finally, a 1*1 convolutional layer and a transposition operation are used to output an N*256 feature map, where N represents the maximum number of human targets identified, which can be adjusted according to actual conditions.

[0139] See also Fig.13 , Fig.13 This is a module diagram of a decoder module provided in an example of the present application.

[0140] In some embodiments, the N*256 feature map is separated row by row to generate a 1*256 feature vector, each of which represents the feature value of a human target. The feature vector is processed by two fully connected layers (including a fully connected operation, a batch normalization operation, and a Relu activation operation) to reduce and increase the dimension of the feature, and finally a 1*2 category vector and a 1*51 key point vector are generated by two independent fully connected operations, where the category vector contains two probability values ​​for describing the probability values ​​of human targets and non-human targets, and the key point vector is converted to 17*3 through a conversion operation, representing the 17 3D key point coordinates (x, y, z) of the human body.

[0141] The posture estimation system based on multimodal fusion of the embodiment of the present application is used to execute the posture estimation method based on multimodal fusion in the above embodiment. Its specific processing process is the same as the posture estimation method based on multimodal fusion in the above embodiment, and will not be repeated here.

[0142] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the posture estimation method based on multimodal fusion in the above embodiment of the present application.

[0143] See also Fig.14 , Fig.14 The hardware structure of the electronic device provided in the embodiment of the present application is illustrated, and the electronic device includes:

[0144] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0145] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the posture estimation method based on multimodal fusion in the embodiment of the present application;

[0146] Input / output interface 903, used to implement information input and output;

[0147] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0148] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0149] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0150] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned posture estimation method based on multimodal fusion is implemented.

[0151] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0153] It can be understood by those skilled in the art that Figure 1-14 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0154] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0155] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0156] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0157] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0162] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A posture estimation method based on multimodal fusion, characterized in that: The method comprises: Acquire an image to be detected, a depth image, and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and an image acquisition device, and the millimeter wave signal data is captured by a sensor; Performing feature projection on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and performing feature projection on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module; Inputting the first feature data and the second feature data into the encoder module for feature fusion, and outputting aggregated feature data; Performing feature separation on the aggregated feature data by the decoder module to obtain a category vector and a key point vector; The target posture of the target object is estimated according to the category vector and the key point vector.

2. The posture estimation method based on multimodal fusion according to claim 1, characterized in that: The deep neural network includes a first feature projection module and a second feature projection module; the feature projection is performed on the image to be detected and the depth image by a preset deep neural network to obtain the first feature data, and the feature projection is performed on the millimeter wave signal data to obtain the second feature data, including: Inputting the image to be detected and the depth image into the first feature projection module, so that the first feature projection module performs a convolution operation on the image to be detected and the depth image, and outputs first feature data; The millimeter wave signal data is input into the second feature projection module, so that the second feature projection module performs a convolution operation on the millimeter wave signal data and outputs second feature data.

3. The posture estimation method based on multimodal fusion according to claim 1, characterized in that: The encoder module includes a first encoder module and a second encoder module; the first feature data and the second feature data are input into the encoder module for feature fusion, and the aggregated feature data is output, including: Inputting the first feature data and the second feature data into the first encoder module, so that the first encoder module merges the first feature data and the second feature data to obtain a merged feature map; Performing feature aggregation on the merged feature map by the first encoder module to obtain a feature aggregation map; The feature aggregation graph is input into the second encoder module, so that the second encoder module performs dimensional feature weighting on the feature aggregation graph and outputs aggregated feature data.

4. The posture estimation method based on multimodal fusion according to claim 1, characterized in that: The decoder module includes a first fully connected layer and a second fully connected layer; the decoder module performs feature separation on the aggregated feature data to obtain a category vector and a key point vector, including: Separating the aggregated features row by row through the decoder module to obtain intermediate features; Inputting the intermediate features into the first fully connected layer for dimensionality reduction processing, and outputting the reduced dimensionality features; Input the reduced-dimensionality feature into the second fully connected layer for dimensionality increase processing, and output the target feature; The decoder module performs a full connection operation on the target feature to generate a category vector and a key point vector.

5. The posture estimation method based on multimodal fusion according to claim 1, characterized in that: The method further comprises: The deep neural network is trained for key point prediction using a preset point-to-point mean square error loss function.

6. The posture estimation method based on multimodal fusion according to claim 3, characterized in that: The first encoder module performs feature aggregation on the merged feature map to obtain Feature aggregation graph, including: Performing an averaging operation on the row features of the merged feature map through the first encoder module to obtain a first weight vector; The features of the merged feature map are weighted in the column dimension according to the first weight vector to obtain a feature aggregation map.

7. The posture estimation method based on multimodal fusion according to claim 3, characterized in that: The step of inputting the feature aggregation graph into the second encoder module so that the second encoder module performs dimensional feature weighting on the feature aggregation graph and outputs aggregated feature data includes: Performing a transposition operation on the feature aggregation graph through the second encoder module to flip the row features and column features of the feature aggregation graph to obtain flipped intermediate features; Performing feature aggregation on the flipped intermediate features through the second encoder module to obtain a second weight vector; The flipped intermediate features are weighted in row dimension according to the second weight vector to obtain aggregated feature data.

8. A posture estimation system based on multimodal fusion, characterized in that: The system comprises: A data acquisition module, used to acquire an image to be detected, a depth image and millimeter wave signal data, wherein the image to be detected includes at least one target object, the depth image is an image used to characterize the distance between the target object and the image acquisition device, and the millimeter wave signal data is captured by a sensor; A feature projection module, used to perform feature projection on the image to be detected and the depth image through a preset deep neural network to obtain first feature data, and perform feature projection on the millimeter wave signal data to obtain second feature data, wherein the deep neural network includes an encoder module and a decoder module; a feature fusion module, configured to input the first feature data and the second feature data into the encoder module for feature fusion, and output aggregated feature data; A feature separation module, used for performing feature separation on the aggregated feature data through the decoder module to obtain a category vector and a key point vector; A posture estimation module is used to estimate the target posture of the target object according to the category vector and the key point vector.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the posture estimation method based on multimodal fusion as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the posture estimation method based on multimodal fusion as described in any one of claims 1 to 7.

Images