Multi-task robot sensing method based on deep learning
By building a multi-task model based on HRNet network in the robot system, the problems of high cost of lidar and limited computing power are solved, low-cost and efficient multi-task perception are achieved, and the robot's perception ability and reasoning speed are improved.
Patent Information
- Application Number
- CN202510667082.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-08-19
AI Technical Summary
In the prior art, the perception cost of lidar is high and the computing power of the embedded robot system is limited, which affects the perception accuracy and real-timeness.
A skeleton network is built based on HRNet network, and human pose estimation, object detection and instance segmentation branches are added to build a multi-task model, and a lightweight and efficient multi-task robot perception method is obtained through image data training.
While reducing costs, the robot's perception ability is improved, human body posture estimation, object detection and instance segmentation are realized, simplifying posture estimation of key points throughout the body, improving inference speed and alleviating the problem of missing detection of humans under the robot's fish-eye camera.
Smart Images
Figure CN120503197A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of robotics technology, and specifically relates to a multi-task robot perception method based on deep learning. Background Art
[0002] The robotics industry has experienced rapid growth, driven by the transformation and upgrading of traditional manufacturing. Scene perception is fundamental to robotic systems, primarily aiming to obtain information about the position and category of objects in space. This information plays a crucial role in guiding the robot's subsequent path planning and motion prediction. Currently, there are two main approaches to scene perception: a multi-sensor fusion approach driven by vision, and a LiDAR-based approach supplemented by other sensors.
[0003] Among them, using lidar for robot scene perception has significant advantages. First, lidar can generate a large amount of reliable measurement data, accurate to the centimeter level, and can clearly identify objects. Second, lidar is less susceptible to interference from changes in the external environment and can operate normally in adverse weather and lighting conditions. At the same time, lidar emits light pulses and calculates distance based on round-trip time, detecting pedestrians and objects in the surrounding area, enabling real-time perception of the surrounding environment and making timely judgments. However, lidar also has shortcomings in its application. For example, heavy fog and rain can weaken the intensity of the laser signal, reducing accuracy. Moreover, lidar's point cloud is based on geometry. While it can distinguish shape, it cannot distinguish color and texture, meaning that lidar cannot distinguish between a paper bag and a stone. Lidar is also expensive, with the current market price around $1,000 or even more.
[0004] In recent years, artificial intelligence theory and methods have advanced rapidly, and the field of robotic perception is flooded with various deep learning algorithms. However, the computing power of the embedded controllers typically used in robots is limited, which directly affects the accuracy and real-time performance of perception. Therefore, how to achieve robot scene perception at a low cost within limited computing power has become a pressing issue. Summary of the Invention
[0005] An embodiment of the present invention provides a multi-task robot perception method based on deep learning to solve the problems in the prior art of high cost of using lidar for perception and limited computing power of the robot's embedded system, which affects the accuracy and real-time performance of perception.
[0006] In order to solve the above technical problems, the embodiments of the present invention disclose the following technical solutions:
[0007] One aspect of the present invention provides a multi-task robot perception method based on deep learning, comprising:
[0008] Build a skeleton network based on the HRNet network;
[0009] Adding a human pose estimation branch after the skeleton network;
[0010] Acquire multiple sets of training data and construct a training data set, where each sample in the training data set consists of image data and three labels for the image data, the three labels being a key point label of the lower body, a target detection label, and an instance segmentation label;
[0011] Based on the network with the human pose estimation branch added, the image data of each sample in the training dataset and the corresponding key point labels of the lower body are used as input, and the human pose estimation model is trained using the preset human pose estimation loss function;
[0012] Add an object detection branch to the network that adds the human pose estimation branch;
[0013] Based on the network with two branches added, the image data of each sample in the training dataset and the corresponding human lower body key point labels and target detection labels are used as input. The preset human pose estimation loss function and target detection loss function are used to train a dual-task model for human pose estimation and target detection.
[0014] Add an instance segmentation branch to the network with two branches to obtain a multi-task network;
[0015] Based on the multi-task network, the image data and corresponding instance segmentation label of each sample in the training dataset are taken as input, the parameters of the dual-task model of human pose estimation and target detection are frozen, and the preset instance segmentation loss function is used to train the multi-task model of human pose estimation, target detection and instance segmentation.
[0016] Optionally, building a skeleton network based on the HRNet network includes:
[0017] The number of bottlenecks or basic blocks in each module in each stage of the HRNet network is adjusted to two.
[0018] Optionally, the output results of the skeleton network are feature maps with 4 times the step size, 8 times the step size, 16 times the step size and 32 times the step size respectively;
[0019] Add a human pose estimation branch after the skeleton network, including:
[0020] Transposed convolution is used on the 4-fold stride feature map to generate a 2-fold stride feature map;
[0021] The feature maps with 2 times the step size and 4 times the step size are used as the output data of the human pose estimation branch.
[0022] Optionally, obtaining multiple sets of training data and constructing a training data set includes:
[0023] Multiple sets of training data are obtained from the COCO open source dataset, each set of training data includes image data and key point labels of the lower body of the human body, target detection labels and instance segmentation labels of the image data; wherein the key point labels of the lower body of the human body are the coordinate information of the left hip, right hip, left knee, right knee, left ankle, right ankle, left big toe, left little toe, right big toe and right little toe of the human body in the image data; the target detection label is the coordinates, width and height of the target center point and the category to which the target belongs; the instance segmentation label is the target mask and the category to which the target belongs.
[0024] Optionally, the network after adding the human body pose estimation branch takes the image data of each sample in the training data set and the corresponding human lower body key point label as input, and uses a preset human body pose estimation loss function to train a human body pose estimation model, including:
[0025] The image data of each sample in the training dataset and the corresponding key point labels of the lower body are used as input to the network with the human pose estimation branch added;
[0026] The following preset human pose estimation loss function is used to train the human pose estimation branch to obtain the human pose estimation model:
[0027] L pose =w hp *L hp +w pull *L pull +w push *L push
[0028] Among them, L pose is the loss function for human posture estimation; L hp is the preset heat map loss function; L pull is the preset intra-class distance loss function; L push is the preset inter-class distance loss function; w hp 、w pull and w push They are the coefficients of the loss functions for balancing human posture estimation, w hp is 1.0, w pull is 0.001, w push is 0.001.
[0029] Optionally, adding a target detection branch to the network to which a human pose estimation branch is added includes:
[0030] The 32-fold step feature map is upsampled by 2 times and concatenated with the 16-fold step feature map to obtain feature map c1;
[0031] The feature map c1 is upsampled by 2 times and concatenated with the feature map with 8 times the step size to obtain the feature map c2;
[0032] The feature map c2 is used as the first output data of the target detection network after convolution with size 1*1;
[0033] Feature map c2 is convolved with feature map c1 to obtain feature map c3.
[0034] Feature map c3 is used as the second output data of the target detection network after convolution with size 1*1;
[0035] Feature map c3 is concatenated with the 32-times-step feature map after 3*3 convolution to obtain feature map c4;
[0036] The feature map c4 is convolved with a size of 1*1 as the third output data of the target detection network.
[0037] Optionally, the network after adding two branches takes the image data of each sample in the training data set and the corresponding human lower body key point label and target detection label as input, and uses the preset human pose estimation loss function and target detection loss function to train a dual-task model of human pose estimation and target detection, including:
[0038] Take the image data of each sample in the training dataset and the corresponding human lower body key point labels and target detection labels as input, and input them into the network after adding the human pose estimation branch and the target detection branch;
[0039] The following preset human pose estimation loss function and target detection loss function are used to train the human pose estimation branch and the target detection branch to obtain a dual-task model of human pose estimation and target detection:
[0040] L pose-det =w pose *L pose +w det *L det
[0041] in,
[0042] L det =w box *L box +w cls *L cls +w obj *L obj
[0043] L detis the target detection loss function; L box Loss function for positioning the preset target frame; L cls is the preset classification loss function; L obj is the preset confidence loss function; w box 、w cls and w obj The coefficients of each loss function for the balance target detection; w box is 0.05, w cls is 0.5, w obj is 1.0;
[0044] w pose and w det are the loss function coefficients for balancing the dual tasks of human pose estimation and target detection, w pose is 1.0,w det 0.001; L pose-det Corresponding loss function for the dual-task model of human pose estimation and target detection.
[0045] Optionally, the example segmentation branch is added to the network with two branches to obtain a multi-task network, including:
[0046] Fuse feature maps with 8x steps, 16x steps, and 32x steps;
[0047] Build mask branches and instance branches on the fused feature map, where the mask branch consists of 4 3*3 convolutions and 1 1*1 convolution; the instance branch consists of 4 3*3 convolutions and 1 3*3 group convolution with 4 groups;
[0048] The instance branch is followed by a fully connected layer to generate instance features, and the instance features are followed by three fully connected layers to output category, confidence, and kernel respectively;
[0049] Obtain the results of the fused feature map in the mask branch and instance branch respectively;
[0050] The instance segmentation result obtained by fusing the two results is used as the output data of the instance segmentation network.
[0051] Optionally, the fusing of feature maps with 8 times the step size, 16 times the step size, and 32 times the step size includes:
[0052] The feature map a1 is obtained by upsampling the 32-fold step length feature map by 2 times and adding it to the 16-fold step length feature map.
[0053] The feature map a1 is upsampled by 2 times and added to the 8-times step feature map to obtain the feature map a2;
[0054] The 32-fold step feature map is upsampled by 4 times to obtain the feature map up1, the feature map a1 is upsampled by 2 times to obtain the feature map up2, and the feature map a2, the feature map up1 and the feature map up2 are concatenated to obtain the fused feature map.
[0055] Optionally, the multi-task network takes the image data and corresponding instance segmentation label of each sample in the training data set as input, freezes the parameters of the dual-task model of human pose estimation and target detection, and uses a preset instance segmentation loss function to train a multi-task model of human pose estimation, target detection and instance segmentation, including:
[0056] Freeze the parameters of the dual-task model for human pose estimation and object detection;
[0057] The image data and corresponding instance segmentation labels of each sample in the training dataset are used as input to the multi-task network. The instance segmentation branch is trained using the following preset instance segmentation loss function to obtain a multi-task model for human pose estimation, object detection, and instance segmentation:
[0058] L inst =w label *L label +w mask *L mask +w dice *L dice +w obj *L obj
[0059] Among them, L inst is the instance segmentation loss function; L label is the preset classification loss function; L mask is the preset pixel classification loss function; L obj is the preset confidence loss function; L dice is the preset similarity loss function; w label 、w mask 、w dice and w obj To balance the coefficients of each loss function for instance segmentation, w label is 2.0, w mask is 5.0; w dice is 2.0; w obj is 1.0.
[0060] The multi-task robot perception method based on deep learning provided by the embodiment of the present invention adopts an image-based deep learning perception method, which greatly saves costs compared with the high-cost lidar perception method.
[0061] Furthermore, the embodiments of the present invention utilize a lightweight and efficient network to share feature extraction information, enhancing the robot's perception capabilities, even when the robot's computing power is limited. This multi-task model combines human pose estimation, object detection, and instance segmentation, thereby improving the robot's perception capabilities. Human pose estimation can determine the presence of humans around the robot and their motion, such as foot direction; object detection can determine the presence of obstacles; and instance segmentation can help the robot better locate itself and handle obstacle occlusion.
[0062] In addition, the network used in the embodiment of the present invention simplifies the posture estimation of key points of the whole body, improves the reasoning speed, and alleviates the problem of missed detection of people under the robot fisheye camera. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 A flowchart of a multi-task robot perception method based on deep learning provided by an embodiment of the present invention;
[0064] Figure 2 A schematic diagram of the composition of a multi-task network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0065] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0066] Figure 1 A flowchart of a multi-task robot perception method based on deep learning provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes the following steps:
[0067] Step S101: Building a skeleton network based on the HRNet network.
[0068] The HRNet (High-ResolutionNet) network is a mature network proposed for the task of human posture estimation. In the embodiment disclosed in the present invention, the high-resolution HRNet network is used as the basis to build a skeleton network.
[0069] The HRNet network consists of four stages. The first stage has one module, each of which consists of four bottleneck blocks; the second stage has one module, each of which consists of four basic blocks; the third stage has four modules, each of which consists of four basic blocks; and the fourth stage has three modules, each of which consists of four basic blocks.
[0070] Since the computing power of the robot embedded system is limited, in the embodiment disclosed in the present invention, the number of bottlenecks or basics in each module in each stage of the HRNet network is adjusted to two to obtain a lightweight and efficient skeleton network.
[0071] In the disclosed embodiment of the present invention, a skeleton network can be used to extract input features, and the output results are feature maps with 4 times the step size, 8 times the step size, 16 times the step size, and 32 times the step size, respectively.
[0072] Step S102: Add a human pose estimation branch after the skeleton network.
[0073] In one embodiment disclosed in the present invention, this step can be implemented in the following manner:
[0074] (1) Transposed convolution is used on the 4-times stride feature map to generate a 2-times stride feature map.
[0075] (2) The generated 2x step length feature map and 4x step length feature map are used as the output data of the human posture estimation branch.
[0076] Therefore, the current network contains a newly added human pose estimation branch.
[0077] Step S103: Acquire multiple sets of training data and construct a training data set.
[0078] In one embodiment disclosed in the present invention, the training data is obtained in the following manner:
[0079] Multiple sets of training data are obtained from the COCO open source dataset. Each set of training data includes image data and key point labels of the lower body of the human body, target detection labels, and instance segmentation labels of the image data. Among them, the key point labels of the lower body of the human body are the coordinate information of the left hip, right hip, left knee, right knee, left ankle, right ankle, left big toe, left little toe, right big toe, and right little toe of the human body in the image data; the target detection label is the coordinates, width and height of the target center point, and the category to which the target belongs; the instance segmentation label is the target mask and the category to which the target belongs.
[0080] The data obtained above is used to construct a training dataset. Each sample in the training dataset consists of image data and three labels for the image data: the key point label of the lower body, the object detection label, and the instance segmentation label.
[0081] Step S104: Based on the network with the human body pose estimation branch added, the image data of each sample in the training data set and the corresponding key point label of the lower body are used as input, and a human body pose estimation model is obtained by training using a preset human body pose estimation loss function.
[0082] In the embodiment disclosed in the present invention, the image data of each sample in the training data set and the corresponding key point labels of the lower body of the human body are used as input and input into the network after adding the human posture estimation branch.
[0083] The following preset human pose estimation loss function is used to train the human pose estimation branch to obtain the human pose estimation model:
[0084] L pose =w hp *L hp +w pull *L pull +w push *L push
[0085] Among them, L pose is the loss function for human posture estimation; L hp is the preset heat map loss function; L pull is the preset intra-class distance loss function; L push is the preset inter-class distance loss function; w hp 、w pull and w push They are the coefficients of the loss functions for balancing human posture estimation, w hp is 1.0, w pull is 0.001, w push is 0.001.
[0086] Step S105: adding a target detection branch to the network to which the human pose estimation branch is added.
[0087] In one embodiment disclosed in the present invention, the target detection branch can be added in the following manner:
[0088] (1) The 32-fold step length feature map corresponding to the open source data of the human key points is upsampled by 2 times and concatenated with the 16-fold step length feature map to obtain the feature map c1.
[0089] (2) The feature map c1 is upsampled by 2 times and concatenated with the 8-fold step feature map corresponding to the open source data of the human key points to obtain the feature map c2.
[0090] (3) The feature map c2 is used as the first output data of the target detection network after convolution with a size of 1*1, that is, the feature map c2 is the first output data output by the target detection network based on the open source data of the set of human key points.
[0091] (4) Feature map c2 undergoes a 3*3 convolution and is concatenated with feature map c1 to obtain feature map c3.
[0092] (5) The feature map c3 is used as the second output data of the target detection network after convolution with a size of 1*1. That is, the feature map c3 is the second output data output by the target detection network based on the open source data of the set of human key points.
[0093] (6) Feature map c3 undergoes 3*3 convolution and is concatenated with the 32-fold step feature map to obtain feature map c4.
[0094] (7) The feature map c4 is used as the third output data of the target detection network after convolution of size 1*1, that is, the feature map c4 is the third output data output by the target detection network based on the open source data of the group of human key points.
[0095] As a result, the current network contains the newly added human pose estimation branch and target detection branch.
[0096] Step S106: Based on the network with two branches added, the image data of each sample in the training data set and the corresponding human lower body key point labels and target detection labels are used as input, and the preset human pose estimation loss function and target detection loss function are used to train a dual-task model of human pose estimation and target detection.
[0097] In one embodiment disclosed in the present invention, a dual-task model of human pose estimation and target detection can be trained in the following manner:
[0098] The image data of each sample in the training dataset and the corresponding human lower body key point labels and target detection labels are used as input to the network after adding the human pose estimation branch and the target detection branch.
[0099] The following preset human pose estimation loss function and target detection loss function are used to train the human pose estimation branch and the target detection branch to obtain a dual-task model of human pose estimation and target detection:
[0100] L pose-det =w pose *L pose +w det *L det
[0101] in,
[0102] L det =w box *L box +w cls *L cls +w obj *L obj
[0103] L det is the target detection loss function; L boxLoss function for positioning the preset target frame; L cls is the preset classification loss function; L obj is the preset confidence loss function; w box 、w cls and w obj The coefficients of each loss function for the balance target detection; w box is 0.05, w cls is 0.5, w obj is 1.0.
[0104] w pose and w det are the loss function coefficients for balancing the dual tasks of human pose estimation and target detection, w pose is 1.0,w det 0.001; L pose-det Corresponding loss function for the dual-task model of human pose estimation and target detection.
[0105] Step S107: Add an instance segmentation branch to the network with two branches to obtain a multi-task network.
[0106] In the embodiment disclosed in the present invention, this step can be implemented in the following manner:
[0107] (1) Fusion of feature maps with 8x, 16x, and 32x strides.
[0108] (2) Build a mask branch and an instance branch on the fused feature map, where the mask branch consists of 4 convolutions of size 3*3 and 1 convolution of size 1*1; the instance branch consists of 4 convolutions of size 3*3 and 1 group convolution of size 3*3 with 4 groups.
[0109] (3) The instance branch is followed by a fully connected layer to generate instance features, and the instance features are followed by three fully connected layers to output category, confidence, and kernel respectively.
[0110] (4) Obtain the results of the fused feature map in the mask branch and instance branch respectively.
[0111] The result of the mask branch is a feature map; the result of the instance branch is an array of category, confidence, and kernel.
[0112] (5) The instance segmentation result obtained by fusing the two results is used as the output data of the instance segmentation network.
[0113] The fusion method used is: multiply the feature map output by the mask branch and the kernel output by the instance branch, and the generated segmentation mask is the instance segmentation result, which is used as the output data of the instance segmentation network.
[0114] Therefore, if Figure 2 As shown in the figure, the current network includes the newly added human pose estimation branch, target detection branch and instance segmentation branch, and the current network is used as a multi-task network.
[0115] In one embodiment disclosed in the present invention, the following method can be used to fuse feature maps with 8 times the step size, 16 times the step size, and 32 times the step size:
[0116] (1) Upsample the 32x step size feature map by 2 times and add it to the 16x step size feature map to obtain feature map a1.
[0117] (2) Feature map a1 is upsampled by 2 times and added to the feature map with 8 times the step size to obtain feature map a2.
[0118] (3) Upsample the 32-fold step feature map by 4 times to obtain feature map up1, upsample the feature map a1 by 2 times to obtain feature map up2, and concatenate feature map a2, feature map up1, and feature map up2 to obtain the fused feature map.
[0119] Step S108: Based on the multi-task network, the image data and corresponding instance segmentation label of each sample in the training data set are used as input, the parameters of the dual-task model of human pose estimation and target detection are frozen, and the preset instance segmentation loss function is used to train to obtain a multi-task model of human pose estimation, target detection and instance segmentation.
[0120] In the embodiment disclosed in the present invention, this step is completed in the following manner:
[0121] Freeze the parameters of the dual-task model of human pose estimation and target detection, turn off the gradient of the parameters of the dual-task model of human pose estimation and target detection, and these parameters will not be updated during back propagation, which has the effect of freezing.
[0122] The image data and corresponding instance segmentation labels of each sample in the training dataset are used as input to the multi-task network. The instance segmentation branch is trained using the following preset instance segmentation loss function to obtain a multi-task model for human pose estimation, object detection, and instance segmentation:
[0123] L inst =w label *L label +w mask *L mask +w dice *L dice +w obj *L obj
[0124] Among them, L inst is the instance segmentation loss function; L label is the preset classification loss function; Lmask is the preset pixel classification loss function; L obj is the preset confidence loss function; L dice is the preset similarity loss function; w label 、w mask 、w dice and w obj To balance the coefficients of each loss function for instance segmentation, w label is 2.0, w mask is 5.0; w dice is 2.0; w obj is 1.0.
[0125] A multi-task model using human pose estimation, target detection, and instance segmentation can obtain human pose estimation, target detection, and instance segmentation results based on the image data subsequently collected by the robot. Human pose estimation can determine whether there are people around the robot and judge the state of human movement, such as the direction of the feet; target detection can determine whether there are obstacles around the robot; instance segmentation can help the robot better locate and better deal with obstacle occlusion problems.
[0126] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A multi-task robot perception method based on deep learning, characterized in that: include: Build a skeleton network based on the HRNet network; Adding a human pose estimation branch after the skeleton network; Acquire multiple sets of training data and construct a training data set, where each sample in the training data set consists of image data and three labels for the image data, the three labels being a key point label of the lower body, a target detection label, and an instance segmentation label; Based on the network with the human pose estimation branch added, the image data of each sample in the training dataset and the corresponding key point labels of the lower body are used as input, and the human pose estimation model is trained using the preset human pose estimation loss function; Add an object detection branch to the network that adds the human pose estimation branch; Based on the network with two branches added, the image data of each sample in the training dataset and the corresponding human lower body key point labels and target detection labels are used as input. The preset human pose estimation loss function and target detection loss function are used to train a dual-task model for human pose estimation and target detection. Add an instance segmentation branch to the network with two branches to obtain a multi-task network; Based on the multi-task network, the image data and corresponding instance segmentation label of each sample in the training dataset are taken as input, the parameters of the dual-task model of human pose estimation and target detection are frozen, and the preset instance segmentation loss function is used to train the multi-task model of human pose estimation, target detection and instance segmentation.
2. The method according to claim 1, characterized in that The skeleton network is constructed based on the HRNet network, including: The number of bottlenecks or basic blocks in each module in each stage of the HRNet network is adjusted to two.
3. The method according to claim 1, characterized in that The output results of the skeleton network are feature maps with 4 times the step size, 8 times the step size, 16 times the step size and 32 times the step size respectively; Add a human pose estimation branch after the skeleton network, including: Transposed convolution is used on the 4-fold stride feature map to generate a 2-fold stride feature map; The feature maps with 2 times the step size and 4 times the step size are used as the output data of the human pose estimation branch.
4. The method according to claim 1, wherein The step of obtaining multiple sets of training data and constructing a training data set includes: Multiple sets of training data are obtained from the COCO open source dataset, each set of training data includes image data and key point labels of the lower body of the human body, target detection labels and instance segmentation labels of the image data; wherein the key point labels of the lower body of the human body are the coordinate information of the left hip, right hip, left knee, right knee, left ankle, right ankle, left big toe, left little toe, right big toe and right little toe of the human body in the image data; the target detection label is the coordinates, width and height of the target center point and the category to which the target belongs; the instance segmentation label is the target mask and the category to which the target belongs.
5. The method according to claim 3, characterized in that The network with the human body pose estimation branch added takes the image data of each sample in the training data set and the corresponding key point label of the lower body as input, and uses a preset human body pose estimation loss function to train a human body pose estimation model, including: The image data of each sample in the training dataset and the corresponding key point labels of the lower body are used as input to the network with the human pose estimation branch added; The following preset human pose estimation loss function is used to train the human pose estimation branch to obtain the human pose estimation model: L pose =w hp *L hp +w pull *L pull +w push *L push Among them, L pose is the loss function for human posture estimation; L hp is the preset heat map loss function; L pull is the preset intra-class distance loss function; L push is the preset inter-class distance loss function; w hp 、w pull and w push They are the coefficients of the loss functions for balancing human posture estimation, w hp is 1.0, w pull is 0.001, w push is 0.
001.
6. The method according to claim 5, characterized in that Adding a target detection branch to the network to which a human pose estimation branch is added includes: The 32-fold step feature map is upsampled by 2 times and concatenated with the 16-fold step feature map to obtain feature map c1; The feature map c1 is upsampled by 2 times and concatenated with the feature map with 8 times the step size to obtain the feature map c2; The feature map c2 is used as the first output data of the target detection network after convolution with size 1*1; Feature map c2 is convolved with feature map c1 to obtain feature map c3. Feature map c3 is used as the second output data of the target detection network after convolution with size 1*1; Feature map c3 is concatenated with the 32-times-step feature map after 3*3 convolution to obtain feature map c4; The feature map c4 is convolved with a size of 1*1 as the third output data of the target detection network.
7. The method according to claim 6, characterized in that The network with two branches added takes the image data of each sample in the training data set and the corresponding human lower body key point labels and target detection labels as input, and uses the preset human pose estimation loss function and target detection loss function to train a dual-task model of human pose estimation and target detection, including: Take the image data of each sample in the training dataset and the corresponding human lower body key point labels and target detection labels as input, and input them into the network after adding the human pose estimation branch and the target detection branch; The following preset human pose estimation loss function and target detection loss function are used to train the human pose estimation branch and the target detection branch to obtain a dual-task model of human pose estimation and target detection: L pose-det =w pose *L pose +w det *L det in, L det =w box *L box +w cls *L cls +w obj *L obj L det is the target detection loss function; L box Loss function for positioning the preset target frame; L cls is the preset classification loss function; L obj is the preset confidence loss function; w box 、w cls and w obj The coefficients of each loss function for the balance target detection; w box is 0.05, w cls is 0.5, w obj is 1.0; w pose and w det are the loss function coefficients for balancing the dual tasks of human pose estimation and target detection, w pose is 1.0,w det 0.001; L pose-det Corresponding loss function for the dual-task model of human pose estimation and target detection.
8. The method according to claim 7, characterized in that The instance segmentation branch is added to the network with two branches to obtain a multi-task network, including: Fuse feature maps with 8x steps, 16x steps, and 32x steps; Build mask branches and instance branches on the fused feature map, where the mask branch consists of 4 3*3 convolutions and 1 1*1 convolution; the instance branch consists of 4 3*3 convolutions and 1 3*3 group convolution with 4 groups; The instance branch is followed by a fully connected layer to generate instance features, and the instance features are followed by three fully connected layers to output category, confidence, and kernel respectively; Obtain the results of the fused feature map in the mask branch and instance branch respectively; The instance segmentation result obtained by fusing the two results is used as the output data of the instance segmentation network.
9. The method according to claim 8, characterized in that The feature map of the fusion of 8 times step size, 16 times step size and 32 times step size includes: The feature map a1 is obtained by upsampling the 32-fold step length feature map by 2 times and adding it to the 16-fold step length feature map. The feature map a1 is upsampled by 2 times and added to the 8-times step feature map to obtain the feature map a2; The 32-fold step feature map is upsampled by 4 times to obtain the feature map up1, the feature map a1 is upsampled by 2 times to obtain the feature map up2, and the feature map a2, the feature map up1 and the feature map up2 are concatenated to obtain the fused feature map.
10. The method according to claim 9, characterized in that The multi-task network takes the image data and corresponding instance segmentation label of each sample in the training dataset as input, freezes the parameters of the dual-task model of human pose estimation and target detection, and uses a preset instance segmentation loss function to train a multi-task model of human pose estimation, target detection and instance segmentation, including: Freeze the parameters of the dual-task model for human pose estimation and object detection; The image data and corresponding instance segmentation labels of each sample in the training dataset are used as input to the multi-task network. The instance segmentation branch is trained using the following preset instance segmentation loss function to obtain a multi-task model for human pose estimation, object detection, and instance segmentation: L inst =w label *L label +w mask *L mask +w dice *L dice +w obj *L obj Among them, L inst is the instance segmentation loss function; L label is the preset classification loss function; L mask is the preset pixel classification loss function; L obj is the preset confidence loss function; L dice is the preset similarity loss function; w label 、w maxk 、w dice and w obj To balance the coefficients of each loss function for instance segmentation, w label is 2.0, w mask is 5.0; w dice is 2.0; w obj is 1.0.