Multi-task detection method, and training method and device of multi-task detection model

Abstract

The embodiment of the invention discloses a multi-task detection method, a multi-task detection model training method and a multi-task detection model training device. A plurality of feature maps with different scales can be obtained through a feature extraction network of the multi-task detection model and are respectively input into a task head network in the corresponding multi-task detection module; various prediction results corresponding to each portrait in the to-be-detected image are obtained through prediction branches in the task head network based on the feature map, and the prediction branches comprise a human body frame prediction branch, a human body key point prediction branch and a behavior prediction branch; various prediction results obtained through prediction based on the same anchor point in the feature map in the task head network processing process correspond to the same portrait; and obtaining a final detection result of the corresponding portrait according to each prediction result corresponding to the same portrait output by each task head network. According to the technical means, the technical problem that the reasoning time is long when a multi-stage detection method is adopted for detection is solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a multi-task detection method, a training method for a multi-task detection model, and an apparatus. Background Technology

[0002] In teaching settings, student behavior recognition is a crucial component of the teaching evaluation system. It can not only analyze students' participation in class to improve their learning motivation, but also analyze teachers' control over the classroom and the classroom atmosphere, helping teachers improve their teaching methods and enhance teaching quality.

[0003] In related technologies, a multi-stage detection method is generally used to identify student classroom behavior. In this method, the first stage uses a neural network model to process classroom images (which can be images from a video recording of students listening to a lecture) to obtain individual student images. The second stage uses another neural network model to process these images to obtain key points about the student's body. Then, another neural network model uses these key points to determine the student's posture or behavior, thus achieving behavior detection. Currently, if behavior recognition also includes head orientation, the second stage requires another neural network model to process the individual student images to obtain the student's head portion. Then, another neural network model uses this head portion to determine the head orientation. In other words, each neural network model in the multi-stage detection method independently completes one task. In this case, the number of inferences for each neural network model in the second stage is equal to the number of individual student images obtained in the first stage. For example, if the number of individual student images in the first stage is n, then each model in the second stage needs to run n inferences. Due to the large number of students, this results in a long inference time. Summary of the Invention

[0004] This application provides a multi-task detection method, a training method and device for a multi-task detection model, to solve the technical problem of long inference time when using multi-stage detection methods in related technologies.

[0005] In a first aspect, one embodiment of this application provides a multi-task detection method, including:

[0006] Acquire the image to be detected;

[0007] The image to be detected is input into a multi-task detection model, which includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The behavior prediction branch is used to predict behaviors under multiple behavior categories.

[0008] The feature extraction network obtains multiple feature maps of different scales based on the image to be detected, and each feature map is input into the corresponding task head network, with one task head network corresponding to one feature map;

[0009] Each prediction branch in the task head network processes the corresponding feature map to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human key point prediction results, and behavior prediction results corresponding to each behavior category. The various prediction results obtained by the task head network based on the same anchor point in the feature map during the processing correspond to the same human figure.

[0010] The final detection result of the corresponding human image in the image to be detected is obtained based on the prediction results of the same human image output by each task head network, and each prediction result corresponds to one final detection result.

[0011] The above describes a method that, by inputting the image to be detected into a multi-task detection model, the model's feature extraction network obtains multiple feature maps at different scales. Each task head network of the multi-task detection model can then process one feature map, and each prediction branch in the task head network can obtain a corresponding prediction result based on the feature map, thus achieving multi-task detection. Currently, the prediction results include at least human bounding box prediction results, human keypoint prediction results, and behavior prediction results for each behavior category. Then, the final detection result for each task is obtained based on the multiple prediction results corresponding to each task. This technique solves the technical problem of long inference time when using multi-stage detection methods in related technologies. Furthermore, compared to the models used in multi-stage methods, the multi-task detection model has a smaller footprint. Moreover, since directly predicting human bounding boxes, human keypoints, and behaviors avoids object ambiguity and severe coupling between models, and during the task head network processing, prediction results based on the same anchor point correspond to the same human image, eliminating the need for prediction result matching and improving the processing efficiency of the multi-task detection model.

[0012] In one embodiment of this application, the plurality of prediction branches further includes at least one of a human body prediction branch and a head bounding box prediction branch; correspondingly, the plurality of prediction results also include at least one of a human body prediction result and a head bounding box prediction result.

[0013] The behavior prediction branch is also used to predict occluded human key points, and correspondingly, the multiple prediction results also include human key point occlusion prediction results.

[0014] As mentioned above, the prediction branches in the task head network can be added or removed according to the actual situation without modifying other structures of the multi-task detection model. For example, human prediction branches and head bounding box prediction branches can be added to achieve human detection and head bounding box detection, which can enrich the types of tasks that the multi-task detection model can handle and make the application scenarios of the multi-task detection model wider.

[0015] In one embodiment of this application, it further includes:

[0016] When the behavior prediction branch in the task head network processes the corresponding feature map, the key point features obtained by the human key point prediction branch in the same task head network based on the feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the feature map, so as to obtain the behavior prediction result corresponding to each behavior category based on the concatenated features.

[0017] As mentioned above, the key point features obtained from the human body key point prediction branch can be used by the behavior prediction branch, which can further improve the accuracy of behavior prediction.

[0018] In one embodiment of this application, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input feature map. The first regression module is used to regress the human bounding box variables to obtain the human bounding box prediction result. The first regression module uses distributed focus loss.

[0019] The human key point prediction branch consists of a second convolution module and a second regression module. The second convolution module is used to predict the human key point variables corresponding to each human image based on the input feature map. The second regression module is used to regress the human key point variables to obtain the human key point prediction result. The second regression module adopts bidirectional distributed focus loss.

[0020] When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each portrait based on the input feature map. The third regression module is used to regress the head bounding box variables to obtain the head bounding box prediction result. The third regression module adopts a bidirectional distributed focus loss.

[0021] As mentioned above, a regression module has been added to predict the human body bounding box, the human head bounding box, and human body key points. This module treats the predicted target location as a distribution and performs regression, which can improve the flexibility and accuracy of the prediction.

[0022] In one embodiment of this application, the bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization digits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is a human keypoint variable, and the variable to be regressed corresponding to the third regression module is a head frame variable. The bias coefficient is determined according to the discretization digits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is a human keypoint variable, the result corresponds to a human keypoint prediction result. When the variable to be regressed is a head frame variable, the result corresponds to a head frame prediction result.

[0023] The above describes the design of a two-way DFL. By setting the bias coefficient, it can be ensured that the regression result is positive when regressing the head frame variable or human key point variable, which facilitates subsequent processing. Furthermore, by setting the granularity of the prediction, the accuracy of the regression result can be improved.

[0024] Secondly, one embodiment of this application also provides a method for training a multi-task detection model, including:

[0025] Obtain a training dataset containing multiple training images. Each portrait in each training image has multiple corresponding ground truth labels, including human bounding box labels, human keypoint labels, and behavior labels under multiple behavior categories.

[0026] The training images are input into a multi-task detection model, which includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and an action prediction branch.

[0027] The feature extraction network obtains multiple training feature maps of different scales based on the training image, and each training feature map is input into the corresponding task head network, with one task head network corresponding to one training feature map;

[0028] Each prediction branch in the task head network processes the corresponding training feature map to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. The various training results predicted based on the same anchor point in the training feature map during the task head network processing correspond to the same human image.

[0029] Based on the various training results and the real labels corresponding to each training result, a variety of training losses are constructed for the corresponding task head network. The various training losses include human body bounding box loss, human body keypoint loss, and behavior loss corresponding to each of the behavior categories.

[0030] The model parameters of the multi-task detection model are updated based on the various training losses corresponding to each task head network, so as to train the multi-task detection model.

[0031] The above describes a multi-task detection model designed to perform at least human bounding box detection, human keypoint detection, and behavior detection. Furthermore, by inputting training images into this model, multi-stage detection is unnecessary; that is, a single-person image is not required before other detections. Multiple tasks can be performed in a single stage (using one model), avoiding the long inference time associated with multi-stage methods. Moreover, the multi-task detection model has a smaller footprint compared to the multiple models used in multi-stage methods. Additionally, since it directly predicts human bounding boxes, human keypoints, and behaviors, end-to-end training is achieved, avoiding object ambiguity and severe coupling between models. Furthermore, during task head network processing, training results predicted based on the same anchor point correspond to the same human image, eliminating the need for matching training results and improving the processing efficiency of the multi-task detection model.

[0032] In one embodiment of this application, the plurality of prediction branches further includes at least one of a human prediction branch and a head bounding box prediction branch. Correspondingly, the plurality of ground truth labels also include at least one of a human label and a head bounding box label. The plurality of training results also include at least one of a human training result and a head bounding box training result. The plurality of training losses also include at least one of a human loss and a head bounding box loss.

[0033] The behavior prediction branch is also used to predict occluded human keypoints. Accordingly, the multiple ground truth labels also include human keypoint occlusion labels, the multiple training results also include human keypoint occlusion training results, and the multiple training losses also include human keypoint occlusion losses.

[0034] As mentioned above, the prediction branches in the task head network can be added or removed according to the actual situation without modifying other structures of the multi-task detection model. For example, human prediction branches and head bounding box prediction branches can be added to achieve human detection and head bounding box detection, which can enrich the types of tasks that the multi-task detection model can handle and make the application scenarios of the multi-task detection model wider.

[0035] In one embodiment of this application, multiple ground truth labels corresponding to the same portrait in the training image are assigned to the same anchor point in the training feature map, and one anchor point is assigned multiple ground truth labels for only one portrait.

[0036] As mentioned above, during the training process, all real labels of the same portrait are assigned to the same anchor point, and one anchor point corresponds to all real labels of one portrait. This shared anchor point mechanism enables end-to-end training, ensuring that the training results obtained based on the same anchor point during the task head network processing correspond to the same portrait. This can improve the accuracy and processing efficiency of the multi-task detection model.

[0037] In one embodiment of this application, the behavior training result corresponding to the behavior category includes the prediction score of the behavior prediction branch for each behavior under the corresponding behavior category, wherein each anchor point corresponding to the same portrait in the training feature map has a corresponding prediction score for each behavior, and correspondingly, the process of determining the behavior loss corresponding to a behavior category includes:

[0038] Based on each predicted score under the behavior category and the corresponding behavior label, the natural log loss corresponding to each predicted score is obtained;

[0039] The natural log loss corresponding to each predicted score is multiplied by the category weight corresponding to the corresponding behavior label to obtain the first multiplication result corresponding to each predicted score. Each behavior under the behavior category has a corresponding category weight. The more behavior labels corresponding to the behavior are in each training image, the smaller the corresponding category weight.

[0040] The first multiplication result corresponding to each predicted score under the same anchor point is added together and then multiplied with the matching score corresponding to the anchor point to obtain the second multiplication result corresponding to the anchor point. The matching score refers to the matching score between the anchor point and the behavior label under the behavior category.

[0041] The sum of the second multiplication results of each anchor point corresponding to the same portrait is added together and then divided by the sum of the matching scores of each anchor point corresponding to the portrait to obtain the behavior loss of the portrait under the behavior category.

[0042] As mentioned above, when predicting behavior, considering the imbalance in the number of different behaviors under the same behavior category and the different matching degrees when matching anchor points and real labels, a behavior loss was designed. That is, the category weight and sample weight (i.e. matching score) are added to the loss function to classify each behavior under each behavior category, which can improve the accuracy of the multi-task detection model.

[0043] In one embodiment of this application, it further includes:

[0044] When the behavior prediction branch in the task head network processes the corresponding training feature map, the key point features obtained by the human key point prediction branch in the same task head network based on the training feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the training feature map, so as to obtain the behavior training result corresponding to each behavior category based on the concatenated features.

[0045] As mentioned above, the key point features obtained from the human body key point prediction branch can be used by the behavior prediction branch, which can further improve the accuracy of behavior prediction.

[0046] In one embodiment of this application, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input training feature map. The first regression module is used to regress the human bounding box variables to obtain the human bounding box training result. The first regression module adopts distributed focus loss.

[0047] The human keypoint prediction branch consists of a second convolution module and a second regression module. The second convolution module is used to predict the human keypoint variables corresponding to each portrait based on the input training feature map. The second regression module is used to regress the human keypoint variables to obtain the human keypoint training results. The second regression module adopts bidirectional distributed focus loss.

[0048] When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each portrait based on the input training feature map. The third regression module is used to regress the head bounding box variables to obtain the head bounding box training results. The third regression module adopts a bidirectional distributed focus loss.

[0049] As mentioned above, a regression module has been added to predict the human body bounding box, the human head bounding box, and human body key points. This module treats the predicted target location as a distribution and performs regression, which can improve the flexibility and accuracy of the prediction.

[0050] In one embodiment of this application, the bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization bits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is a human keypoint variable, and the variable to be regressed corresponding to the third regression module is a head frame variable. The bias coefficient is determined according to the discretization bits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is a human keypoint variable, the result corresponds to the human keypoint training result. When the variable to be regressed is a head frame variable, the result corresponds to the head frame training result.

[0051] The above describes the design of a two-way DFL. By setting the bias coefficient, it is possible to ensure that the regression results for the head frame variable and the human body keypoint variable are positive, which facilitates subsequent processing. Furthermore, by setting the granularity of the prediction, the accuracy of the regression results can be improved.

[0052] Thirdly, one embodiment of this application also provides a multi-task detection device, including:

[0053] The first acquisition unit is used to acquire the image to be detected;

[0054] The first input unit is used to input the image to be detected into a multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches. The multiple prediction branches include a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The behavior prediction branch is used to predict behaviors under multiple behavior categories.

[0055] The first feature extraction unit is used to obtain multiple feature maps of different scales from the image to be detected by the feature extraction network and input each feature map into the corresponding task head network, with one task head network corresponding to one feature map.

[0056] The first prediction unit is used to process the corresponding feature maps by each prediction branch in the task head network to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human key point prediction results, and behavior prediction results corresponding to each behavior category. The various prediction results obtained by the task head network based on the same anchor point in the feature map during the processing correspond to the same human figure.

[0057] The result determination unit is used to obtain the final detection result of the corresponding human image in the image to be detected based on the prediction results of the same human image output by each task head network, wherein each prediction result corresponds to one final detection result.

[0058] Fourthly, one embodiment of this application also provides a training apparatus for a multi-task detection model, comprising:

[0059] The second acquisition unit is used to acquire a training dataset, which contains multiple training images. Each portrait in each training image has multiple corresponding real labels, including human bounding box labels, human key point labels, and behavior labels under multiple behavior categories.

[0060] The second input unit is used to input the training image into a multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and an action prediction branch.

[0061] The second feature extraction unit is used to obtain multiple training feature maps of different scales from the training image by the feature extraction network and input each training feature map into the corresponding task head network, with one task head network corresponding to one training feature map.

[0062] The second prediction unit is used to process the corresponding training feature maps by each prediction branch in the task head network to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. The various training results predicted based on the same anchor point in the training feature map during the task head network processing correspond to the same human image.

[0063] The loss construction unit is used to construct multiple training losses corresponding to the task head network based on the multiple training results and the real labels corresponding to each training result. The multiple training losses include human body bounding box loss, human body keypoint loss, and behavior loss corresponding to each behavior category.

[0064] The parameter update unit is used to update the model parameters of the multi-task detection model based on the multiple training losses corresponding to each task head network, so as to train the multi-task detection model.

[0065] Fifthly, one embodiment of this application also provides a multi-task detection device, including: one or more processors and a memory;

[0066] The memory is used to store one or more programs;

[0067] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task detection method as described in the first aspect.

[0068] Sixthly, one embodiment of this application also provides a training device for a multi-task detection model, including: one or more processors and a memory;

[0069] The memory is used to store one or more programs;

[0070] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the multi-task detection model as described in the second aspect.

[0071] In a seventh aspect, one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the multitasking detection method as described in the first aspect.

[0072] Eighthly, one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the multi-task detection model as described in the second aspect.

[0073] The beneficial effects of the multi-task detection device, equipment, and storage medium provided above can be referenced to the beneficial effects of the multi-task detection method, and the beneficial effects of the multi-task detection model training device, equipment, and storage medium can be referenced to the beneficial effects of the multi-task detection model training method. Attached Figure Description

[0074] Figure 1 This is a flowchart of a multi-stage detection method in related technologies;

[0075] Figure 2 This is a schematic diagram of the detection results of key points on the human body in related technologies.

[0076] Figure 3 A flowchart illustrating a training method for a multi-task detection model provided in one embodiment of this application;

[0077] Figure 4 This is a schematic diagram of the position of a human head frame provided in one embodiment of this application;

[0078] Figure 5 A schematic diagram of a multi-task detection model provided in one embodiment of this application;

[0079] Figure 6A flowchart illustrating a multi-task detection method provided in one embodiment of this application;

[0080] Figure 7 This is a schematic diagram of the structure of a multi-task detection device provided in one embodiment of this application;

[0081] Figure 8 A schematic diagram of the structure of a training device for a multi-task detection model provided in one embodiment of this application;

[0082] Figure 9 This is a schematic diagram of the structure of a multi-task detection device provided in one embodiment of this application;

[0083] Figure 10 This is a schematic diagram of the structure of a training device for a multi-task detection model provided in one embodiment of this application. Detailed Implementation

[0084] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0085] Figure 1 This is a flowchart of a multi-stage detection method in related technologies, for reference. Figure 1 First, human detection is performed on the classroom images. This detection model can be a neural network model. The neural network model identifies each human body in the classroom image (identified by a bounding box), and then crops the classroom image based on these bounding boxes to obtain a single-person image for each body. Currently, since the bounding boxes are rectangular, the single-person images obtained based on these bounding boxes are also rectangular. At this point, the first stage is considered complete. In the second stage, a neural network model is used to detect key points in the single-person images to identify the key points of each human body. Then, another neural network model is used to perform pose or behavior recognition based on the identified key points to obtain the pose or behavior corresponding to the single-person image. Furthermore, if head classification information (such as head tilting, head tilting, head turning to the side, or head turning away) is needed, a neural network model is required to detect heads in single-person images to identify the head portion (identified by a head bounding box). Then, a neural network model is used to classify the head based on this portion (i.e., classifying it as head tilting, head tilting, head turning to the side, or head turning away) to determine the head's orientation. At this point, Figure 1The detection method shown requires multiple neural network models, and each neural network model is used to process one task. That is, each neural network model can be regarded as a single-task neural network model. After the single-task neural network models cooperate with each other, a multi-stage detection method can be realized.

[0086] This multi-stage detection method uses single-task neural network models with clear structures and easy training. However, it has the following problems: 1. Large model size and long inference time; 2. Object ambiguity; 3. Severe coupling between models.

[0087] The problems of large model footprint and long inference time are specifically as follows: The neural network models used in the multi-stage detection method adopt a top-down approach. The number of inferences of the neural network model in the second stage is equal to the number of individual images obtained in the first stage. For example, if the number of individual images obtained in the first stage is n, then each model needs to run inference n times in the second stage. Due to the large number of students, this results in a long inference time. Furthermore, the storage and runtime of multiple neural network models require significant resources, leading to a large model footprint.

[0088] The specific problem of object ambiguity is that students in classroom images are relatively close together. This means that individual images obtained after processing these images often include parts of other people's limbs. When performing human keypoint detection or head detection, misidentification can easily occur; that is, human keypoints or head parts of other people in the individual image may be misidentified as the human keypoints or head parts of the student in the individual image. For example, Figure 2 This is a schematic diagram of the detection results of key points on the human body in related technologies. Figure 2 If a single-person image contains parts of other people's limbs, the body keypoints obtained from that single-person image in the second stage will include the body keypoints of other people. In this case, based on the body keypoints in the single-person image (which include the body keypoints of other people), there is a high probability of obtaining incorrect body postures or behaviors. Thus, when a single-person image contains parts of other people's limbs, it will severely affect the accuracy of subsequent body posture or behavior recognition and head direction recognition, i.e., affect the accuracy of student behavior recognition, and consequently lead to inaccurate teaching evaluation. In related technologies, additional matching and filtering strategies can be added to avoid object ambiguity; however, this increases the computational cost of multi-stage detection methods.

[0089] The problem of severe coupling between models is that the accuracy of subsequent models is limited by the accuracy of previous models. For example, the accuracy of human keypoint detection in the second stage depends on the accuracy of the individual person image. If the individual person image is inaccurate, it will affect the accuracy of subsequent human keypoint detection. When the accuracy of human keypoint detection is affected, it will further affect the accuracy of subsequent body pose or behavior recognition. In this way, end-to-end training and testing cannot be achieved, meaning that the final result cannot be obtained directly from the initial input image.

[0090] To address the problems of the aforementioned multi-stage detection methods, this application provides a multi-task detection model. This model can perform multiple tasks such as human bounding box detection, human keypoint detection, and behavior detection. Each task has a corresponding prediction branch in the multi-task detection model, and the branches have no hierarchical relationship (i.e., no multiple stages). It does not require obtaining a single person image before performing other detections. Furthermore, during the processing of the multi-task detection model, all prediction results (such as human bounding box prediction results, human keypoint prediction results, and behavior prediction results) based on the same anchor point in the feature map correspond to the same human image. That is, each prediction branch uses the same anchor point when predicting the same human image. This enables end-to-end training of the multi-task detection model without the need for multiple other neural network models or a top-down approach. It can simultaneously achieve human bounding box detection, human keypoint detection, and behavior detection, avoiding problems such as object ambiguity, large model footprint and long inference time, and severe coupling between models, resulting in higher detection accuracy and efficiency.

[0091] Based on this, to better understand the usage process of the multi-task detection model, the training process of the multi-task detection model is first introduced. Accordingly, this application provides a training method for a multi-task detection model. This training method can be executed by a training device for the multi-task detection model, which can consist of two or more physical entities, or a single physical entity. Currently, the training device can be a computer (PC), server, or other equipment with data processing capabilities.

[0092] The training device includes at least a processor and memory. The processor and memory can be connected via a bus or other means.

[0093] The number of processors can be one or more. Processors may include processing units such as application processors (APs), graphics processing units (GPUs), and central processing units (CPUs).

[0094] Memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules (units), such as the program instructions / modules corresponding to the training device in the embodiments of this application. Memory may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the training device, etc. Furthermore, memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the training device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] In addition, the training equipment may also include one or more components such as a display screen, a component for accessing a network (such as the Internet), a communication interface, a power supply, a speaker, a camera, and an input device, etc., but the embodiments are not limited thereto.

[0096] Based on the aforementioned hardware structure, the training device is equipped with at least one operating system. This operating system can be Android, Windows, or Linux, among others. Using this installed operating system, the training device can train a multi-task detection model.

[0097] Figure 3 This is a flowchart illustrating a training method for a multi-task detection model according to one embodiment of this application, with reference to... Figure 3 The training method for this multi-task detection model includes steps 110-160:

[0098] Step 110: Obtain the training dataset. The training dataset contains multiple training images. Each portrait in each training image has multiple corresponding ground truth labels, including human bounding box labels, human keypoint labels, and behavior labels under multiple behavior categories.

[0099] The training dataset is the dataset used to train the multi-task detection model. It contains multiple (or a large number) images for training. Currently, the images in the training dataset are referred to as training images. Each training image contains at least one or more human figures, and each human figure has multiple corresponding ground truth labels. Ground truth labels can be understood as known results, and the content of the ground truth labels is related to the type of task the multi-task detection model performs. In one embodiment, the multiple tasks implemented by the multi-task detection model include at least human bounding box (i.e., a rectangular box identifying a human body) detection, human keypoint (i.e., points identifying key parts of the human body) detection, and behavior detection under multiple behavior categories. Accordingly, the ground truth labels should include human bounding box labels, human keypoint labels, and behavior labels under multiple behavior categories. The human bounding box label can clearly indicate the true location of the corresponding human body in the training image. The human keypoint label can clearly indicate the true location of each human keypoint in the training image. It should be noted that which locations in the human body can be used as human keypoints can be set according to the actual situation. For example, in a teaching scenario, the students in the classroom image are mainly upper body, so some parts of the upper body can be used as human keypoints. By using behavior labels under each behavior category, the specific behaviors performed by the human body under various behavior categories can be clearly identified. The behavior categories to be detected can be set according to the actual situation. For example, when recognizing the behavior of students in a teaching scenario, there are three types of behaviors to be detected: sitting or standing, raising or not raising hands, and specific head direction (such as looking up, looking down, turning head to the side, turning head away, and other directions).

[0100] Various real-world labels can be manually added to the training images. For example, manual additions can be made to each portrait in the training images, including rectangular boxes enclosing the body as body bounding boxes, dotted markers at key points as body key point labels, and indications of the body's current behavior (e.g., whether the body is standing or sitting, with hands raised or not, and head direction (head up, head down, head tilted to the side, head turned away, or other direction)). The source of the training images is currently not limited; for example, images from publicly available image sets can be used. Furthermore, when the multi-task detection model is applied to teaching scenarios, the training images can be classroom images captured from a large number of classes. Similarly, when the multi-task detection model is applied to meeting scenarios, the training images can be meeting images captured from a large number of meetings.

[0101] Currently, training devices can directly acquire training datasets with real labels and use these datasets to train multi-task detection models.

[0102] Step 120: Input the training images into the multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and an action prediction branch.

[0103] For example, a multi-task detection model includes a feature extraction network and multiple task head networks. The feature extraction network includes a backbone network and a neck network. The backbone network can be considered the core network for feature extraction; its input is a complete image (e.g., the training image during training), and its output is features. The neck network can be considered the network for feature enhancement; the neck network connects the backbone network and each task head network to complete the feature enhancement task. The multiple task head networks, also referred to as "heads," are used to complete the main tasks, such as bounding box localization and object classification. Currently, the tasks that task head networks can perform include at least human bounding box detection, human keypoint detection, and behavior detection under various behavior categories. Human bounding box detection and human keypoint detection can be considered localization tasks, and behavior detection under each behavior category can be considered object classification tasks.

[0104] Backbone, Neck, and head are commonly used terms in object detection. The neural network model used for object detection typically consists of a backbone, neck, and head, which will not be elaborated upon further here. In one embodiment, to enable the multi-task detection model to address the problems caused by multi-stage detection methods in related technologies and to achieve multi-task detection, a neural network model for object detection composed of a backbone, neck, and head is used. Each task head network in the head is improved to obtain the currently used multi-task detection model. In this case, the backbone and neck networks in the multi-task detection model can refer to existing backbones and necks; this embodiment does not limit their use.

[0105] Optionally, the number of task head networks included in the head is related to the number of feature maps output by the Neck network. It can be understood that the Neck network can output multi-scale feature maps (the specific number can be set according to actual conditions). Each feature map is input to a corresponding task head network, enabling the task head network to complete the corresponding detection task based on the features in the feature map. Currently, each task head network has the same structure and the same processing procedure. Therefore, this embodiment uses a single task head network as an example to describe its structure and processing procedure.

[0106] For example, the task head network can include multiple prediction branches, each performing a different detection task, thereby achieving multi-task detection. In one embodiment, the multiple prediction branches of the task head network include at least a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The human bounding box prediction branch processes the feature map output by the Neck network to detect (i.e., predict) the human bounding box for each portrait. The human keypoint prediction branch processes the feature map output by the Neck network to detect (i.e., predict) the human keypoints for each portrait. The behavior prediction branch processes the feature map output by the Neck network to detect (i.e., predict) the specific behavior of each portrait under each behavior category. Each portrait detected by the multi-task detection model in the input image (e.g., the training image in the training process) has a corresponding human bounding box, human keypoints, and behavior (the features of the portrait can be reflected in the feature map).

[0107] Optionally, in practical applications, new detection tasks can be added based on the specific circumstances, and corresponding prediction branches can be added to the task head network. This allows for relatively flexible addition and removal of prediction branches. For example, a human detection task and / or a head bounding box detection task can be added. In this case, multiple prediction branches can include at least one of a human prediction branch and a head bounding box prediction branch. The human prediction branch processes the feature map output by the Neck network to detect (i.e., predict) the human figures (i.e., detect the human figures appearing in the feature map and their probabilities (i.e., confidence levels)). When multiple prediction branches include a human prediction branch, to ensure accurate prediction, the multiple ground truth labels corresponding to each portrait in the training image also need to include human labels. These labels represent the human figures appearing in the training image. Human labels can also be added manually. In practical applications, the human prediction branches can also share human bounding box labels, meaning the human bounding box labels can also serve as human labels. The head bounding box prediction branch processes the feature map output by the Neck network to detect (i.e., predict) the head bounding boxes of each portrait. When multiple prediction branches include a head bounding box prediction branch, in order to ensure that the head bounding box prediction branch can predict accurately, the multiple ground truth labels corresponding to each portrait in the training image also need to include head bounding box labels. The head bounding box labels can clearly indicate the real position of the head part in the portrait in the training image. That is, the head bounding box labels are used to outline the position of the head part in the training image.

[0108] Optionally, the behavior prediction branch, in addition to detecting various behavior types, can also detect occlusion of human keypoints. This is understandable; when two figures are close together, one or more keypoints in one figure may be occluded by the other. For example, Figure 2When two portraits overlap, portrait 21 occludes part of the key human features of portrait 22. Therefore, in this embodiment, the behavior prediction branch can also be used to predict the occluded key human features in each portrait. To ensure accurate prediction of key human feature occlusion, multiple ground truth labels should also include key human feature occlusion labels, which are used to identify the occluded key human features in the portrait.

[0109] For example, after the training image is input into the multi-task detection model, it can be processed by the feature extraction network and the task head network to obtain the detection result. The processing of the feature extraction network and the task head network can be found in steps 130 and 140.

[0110] Step 130: The feature extraction network obtains multiple training feature maps of different scales from the training images and inputs each training feature map into the corresponding task head network. One task head network corresponds to one training feature map.

[0111] In this multi-task detection model, after the training images are input, the feature extraction network first processes the training images to obtain multiple feature maps at different scales. Specifically, the Backbone network in the feature extraction network first processes the training images to extract features at different scales. Then, the Neck network concatenates and enhances the multi-scale features to obtain multiple enhanced feature maps at different scales. The specific processing procedures of the Backbone and Neck networks can be found in existing documentation. Multi-scale feature extraction is a commonly used technique and will not be elaborated upon here.

[0112] In one embodiment, during the training process, the feature map obtained by the feature extraction network based on the training image is recorded as the training feature map. One training feature map corresponds to one task head network, that is, each task head network can receive a training feature map at one scale.

[0113] Step 140: Each prediction branch in the task head network processes the corresponding training feature map to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. During the task head network processing, the various training results predicted based on the same anchor point in the training feature map correspond to the same human image.

[0114] For example, when the training feature map is input to the task head network, each prediction branch in the task head network can receive the training feature map and obtain the corresponding detection result based on the features represented in the training feature map. In one embodiment, during training, the detection result output by the prediction branch is recorded as the training result. In this case, the task head network can be considered to output multiple training results. Currently, the training result output by the human bounding box prediction branch is the human bounding box training result, which is the position-related representation of the human bounding box in the training feature map; one human bounding box training result corresponds to one human portrait. The training result output by the human keypoint prediction branch is the human keypoint training result, which is the position-related representation of each human keypoint in the human portrait in the training feature map; one human keypoint training result corresponds to one human portrait. The training results output by the behavior prediction branch are the behavior training results for each behavior category of the human image. Each behavior category corresponds to one behavior training result, which represents the probability (i.e., confidence) of the specific behavior being identified. For example, if the behavior categories include raising hands and not raising hands, then the behavior training result output by the behavior prediction branch should indicate whether the identified behavior is raising hands or not raising hands and the corresponding probability. Alternatively, the behavior training result output by the behavior prediction branch should indicate the probability of the identified behavior being raising hands and the probability of the behavior being not raising hands. In this case, each behavior under the behavior category has a corresponding probability of occurrence in the behavior training result.

[0115] Optionally, when multiple prediction branches include at least one of a human body prediction branch and a head bounding box prediction branch, the multiple training results also correspond to at least one of a human body training result and a head bounding box training result. The human body training result is the probability of each human figure in the training feature map (optionally, results with very low probabilities can be deleted). The head bounding box training result is the position-related representation of the head bounding box in the training feature map; one human figure corresponds to one head bounding box training result. Optionally, when the behavior prediction branch is also used to predict occluded human keypoints, the multiple training results also include human keypoint occlusion training results; one human keypoint corresponds to one human keypoint occlusion training result, which is used to identify whether the corresponding human keypoint is occluded (may include the probability of occlusion).

[0116] It is understandable that each human figure in the current training image that can be recognized (i.e. detected) by the multi-task detection model should have a corresponding training result.

[0117] In one embodiment, since the task head network can obtain multiple prediction results for the same image, in order to avoid the need to match multiple prediction results for the same image later (e.g., matching the training results of the human bounding box, the training results of the human keypoints, and the training results of the behavior under each behavior category for the same image to clarify the training results of the human bounding box, the training results of the human keypoints, and the training results of the behavior under each behavior category that the same image should correspond to), the task head network adopts a shared anchor point mechanism during the processing. That is, the ground truth labels corresponding to the same image can be assigned to the same anchor point, and one anchor point is only assigned to the ground truth labels of one image. In other words, multiple ground truth labels corresponding to the same image in the training image are assigned to the same anchor point in the training feature map, and one anchor point is only assigned to multiple ground truth labels of one image.

[0118] In this context, anchor points in the feature map (currently the training feature map) can be understood as pre-defined feature locations on the feature map. For example, a feature point or region in the feature map can serve as an anchor point. In object detection tasks, anchor points provide a reference frame, helping to predict the position and size of target objects (such as the human body, key points of the human body, etc.), thus improving detection efficiency and accuracy. Optionally, the anchor points used by the task head network can be automatically generated, meaning the multi-task detection model automatically generates the anchor points of the training feature map for use by the task head network. The technical implementation for automatically generating anchor points is not currently limited; existing technical implementations can be referenced.

[0119] For example, during training, to assign each ground truth label corresponding to the same image to the same anchor point, the SimOTA method can be used for positive sample allocation. In this case, the successfully assigned ground truth labels and anchor points can be considered positive samples. Specifically, after generating anchor points, the SimOTA method can be used to match anchor points with ground truth labels. SimOTA is a matching strategy that minimizes the cost of assigning each anchor point in the feature map to the ground truth (gt).

[0120] Optionally, when using the SimOTA method, the cost allocation can be calculated primarily based on the human bounding box labels. This means calculating the cost based on anchor points and each human bounding box label in the training image to achieve the allocation of human bounding box labels and anchor points. The aforementioned process is the calculation process of the existing SimOTA method and will not be described again here. During allocation, only one human bounding box label is assigned to each anchor point; that is, when using SimOTA for allocation, the value of dynamic_k (i.e., the number of positive samples) is set to 1. After the allocation of anchor points and human bounding box labels, other ground truth labels belonging to the same image as the human bounding box label (e.g., human keypoint labels and behavior training results corresponding to various behavior categories, as well as head bounding box labels, human body labels, and human keypoint occlusion labels) can be defaulted to the anchor points corresponding to the human bounding box labels to complete the allocation of each ground truth label. Alternatively, when using the SimOTA method, the cost allocation can be calculated primarily based on human bounding box labels and secondarily based on human keypoint labels. That is, the classification loss and regression loss obtained during the cost calculation process consist of the classification loss and regression loss corresponding to the human bounding box and the classification loss and regression loss corresponding to the human keypoint. Next, based on the calculated cost, the body bounding box labels and body keypoint labels for the same person are assigned to one anchor point, and each anchor point is assigned only one person's body bounding box labels and body keypoint labels. Then, other real-world labels belonging to the same portrait as the body bounding box labels are assigned to the anchor points corresponding to those body bounding box labels. It should be noted that SimOTA is an existing technical method, and this embodiment will not elaborate further. It is understood that after allocation in the aforementioned manner, one anchor point corresponds to all real-world labels for one portrait; however, all real-world labels for one portrait can correspond to multiple anchor points.

[0121] In practical applications, other allocation methods can also be used to assign the real labels corresponding to the same person to the same anchor point. For example, the TAL (Task Alignment Learning) allocation strategy can be used, which is a label allocation method used to improve object detection algorithms. This application does not limit the allocation method used for anchor point allocation.

[0122] Understandably, during training, after all the real labels of the same portrait are assigned to the same anchor point, when the prediction branches of the task head network make predictions based on the features represented by the anchor point, it can be guaranteed that all prediction results obtained based on the same anchor point belong to the same person.

[0123] For example, after the anchor points in the training feature map are assigned, each prediction branch of the task head network can output the corresponding training results based on the training feature map. The network structure of each prediction branch in the task head network can be set according to the actual situation to ensure that the corresponding detection task can be completed.

[0124] In one embodiment, the human bounding box prediction branch can be implemented using a convolutional network. In this case, to reduce the error of the human bounding box prediction branch, distribution focal loss (DFL) can be used on top of the convolutional network for prediction; that is, DFL can be used during location prediction.

[0125] DFL (Distributed Boundary Function) is a regression method applied to bounding box regression. It treats the boundaries of the predicted target (currently the bounding boxes of a human body) as a distribution, meaning the positions of the bounding boxes relative to their internal anchor points are considered a distribution. This improves the flexibility and accuracy of bounding box regression. DFL can be expressed by the following formula:

[0126]

[0127] in, The discretized formula, P(i), can be understood as the distribution of positions. Specifically, N represents the number of bits in the discretization, reflecting the maximum scale of the prediction, i.e., the maximum distance during regression. The specific value of N can be set according to the actual situation. Taking human bounding box prediction as an example, N = 16, indicating that the predicted human bounding box position is expressed using a discretized 16-bit vector. i represents the i-th bit in the N-bit vector, P(i) represents the data in the i-th bit of the N-bit vector, and d... t This represents the result of the regression, that is, regressing an N-bit vector into a specific value. Taking human bounding box prediction as an example, d t This represents the distance from a corner point within the predicted human bounding box to an anchor point inside the bounding box. Understandably, d... t The distance is normalized using the downsampling factor of the feature map; therefore, the actual distance should be d. t *s l s l This represents the downsampling factor of the feature map. It can be understood that in a multi-task detection model, when the Neck network outputs multiple training feature maps at different scales, each training feature map has a corresponding downsampling factor (i.e., a downsampling factor relative to the training image). For example, if the downsampling factor is 16, then d obtained based on the training feature map... t It should also be that this downsampling factor was used, that is, d was obtained based on the training feature map downsampled by 16 times. t At this point, in order to ensure that the regressed distance can be applied to the training images, it is also necessary to further refine d. t *s l This is used to obtain the distance applicable to the training images.

[0128] When using DFL for human bounding box prediction, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input training feature map. The first regression module is used to regress the human bounding box variables to obtain the human bounding box training results. The first regression module uses distributed focus loss.

[0129] The first convolutional module can also be considered a convolutional network. Its structure can be customized based on specific circumstances. For example, it can consist of 2×conv3+conv1, where conv3 represents a 3×3 convolution and conv1 represents a 1×1 convolution. Currently, the first convolutional module obtains features related to the human bounding box in the training image based on the features in the input training feature map (i.e., the features represented by each anchor point). Then, based on these features, it predicts the position of the human bounding box for each portrait in the training feature map (this position corresponds to the downsampled position of the training image), and outputs a result representing the human bounding box position. In one embodiment, the output of the first convolutional model is denoted as a human bounding box variable, with one human bounding box variable corresponding to one human bounding box. Currently, for the first regression model to perform regression, the human bounding box variable output by the first convolutional model should be a discretized human bounding box position. In this case, the human bounding box variable can be represented using 4*N bits of data. For example, when representing the position of a human bounding box using its top-left corner (currently denoted as the top-left corner) and bottom-right corner (currently denoted as the bottom-right corner), the 4*N bits of data include N-bit discretized data of the top-left corner's position on the x-axis (this position can be represented by coordinates or distance from the corresponding anchor point), N-bit discretized data of the top-left corner's position on the y-axis, N-bit discretized data of the bottom-right corner's position on the x-axis, and N-bit discretized data of the bottom-right corner's position on the y-axis. Here, the x-axis and y-axis refer to the coordinate axes used for training the feature map, with their origin located at the top-left corner of the training feature map. It can be understood that the human bounding box label can also be represented by the positions of the top-left and bottom-right corners of the human bounding box.

[0130] The first regression model is used to regress the human body frame variable to obtain the human body frame training result, where one human body frame variable corresponds to one human body frame training result. Currently, the first regression model uses Discrete Folded Linear Algorithm (DFL), that is, it uses the aforementioned DFL formula to regress the human body frame variable. Here, the variable to be regressed is the human body frame variable, and P(i) in the formula is the i-th data point in the N-bit discretized data of the human body frame variable, d... t This is the result after regression. For example, when the position of the human body frame is represented by the top left and bottom right corners, after regressing the human body frame variable using the first regression model, four d values ​​can be obtained. tAnd these represent the distances on the x-axis and y-axis respectively between the top-left corner of the human bounding box and its corresponding anchor point (i.e., the anchor point matched by the human bounding box label), and the distances on the x-axis and y-axis respectively between the bottom-right corner of the human bounding box and its corresponding anchor point. Here, the x-axis and y-axis are the coordinate axes used for training the feature map, with their origin located at the top-left corner of the training feature map. The resulting four d values... t This can be considered as a training result of a human bounding box. It can be understood that the downsampling factor corresponding to the training feature map and 4 d... t After multiplying these values ​​separately, we can obtain the distances on the x-axis and y-axis between the top-left corner of the human bounding box and its corresponding anchor point in the training image, as well as the distances on the x-axis and y-axis between the bottom-right corner of the human bounding box and its corresponding anchor point. This allows us to obtain the coordinates of the top-left and bottom-right corners of the human bounding box, thus predicting its location. It can be understood that if the position of the human bounding box is represented by its center point coordinates, width, and height, then the coordinates of the top-left and bottom-right corners of the human bounding box obtained from the first regression module can also yield the coordinates of its center point, width, and height.

[0131] It should be noted that a human bounding box label may match multiple anchor points. In related technologies, the problem of obtaining the final distance based on multiple anchor points during regression has been solved, and the implementation will not be described in detail here.

[0132] When assigning anchor points based primarily on human body bounding box labels according to SimOTA, the matched anchor points should be located within a square with the center point of the human body bounding box as the reference. Therefore, these anchor points will generally also be located within the corresponding human body bounding box. In this case, when using DFL to regress the human body bounding box variable, the obtained d t If the value is positive, it means that the distance from the corner point to the internal anchor point is predicted (which can also be understood as the distance from the four sides of the human body frame to the internal anchor point). This process ignores anchor points located outside the human body frame.

[0133] As is understandable, DFL is a commonly used regression method, and its specific calculation process will not be elaborated here.

[0134] In one embodiment, the human keypoint prediction branch can be implemented using a convolutional network. In this case, to reduce the error of the human keypoint prediction branch, a Direct-Focused Algorithm (DFL) can also be used to predict human keypoints on top of the convolutional network. Currently, because the human keypoint labels and human bounding box labels share the same anchor point, the relative positional relationship between each human keypoint and the anchor point cannot be guaranteed. That is, each human keypoint of a portrait can be located in any direction from the anchor point. Therefore, when using DFL to regress the human keypoint variable, the resulting d... tNegative values ​​are possible, meaning positive distances cannot be obtained. Furthermore, since a human keypoint is located at a single point (i.e., without four sides), its position is more flexible. Therefore, using DFL for prediction also presents limitations in prediction accuracy. It's understandable that by setting the value of N, the maximum scale (maximum number of discretization bits) for regression can be set, but the prediction accuracy cannot be changed. Moreover, the required accuracy for predicting the position of a point is higher than the accuracy required for predicting the position of a bounding box. Therefore, in one embodiment, to avoid the possibility of negative distances after regression and the limitation in accuracy caused by human keypoints being located in any direction from the anchor point, bidirectional DFL is used for regression. In this case, the human keypoint prediction branch consists of a second convolutional module and a second regression module. The second convolutional module predicts the human keypoint variables corresponding to each portrait based on the input training feature map. The second regression module regresses the human keypoint variables to obtain the human keypoint training results. The second regression module uses bidirectional distributed focal loss (i.e., bidirectional DFL, also denoted as BS-DFL).

[0135] The second convolutional module can also be considered a convolutional network. Its structure can be customized based on specific circumstances; for example, it can consist of 2×conv3+conv3+conv1. Currently, the second convolutional module, based on the features in the input training feature map (i.e., the features represented by each anchor point), can obtain the features of each human keypoint belonging to the same portrait in the training image. Then, based on the obtained features, it predicts the human keypoints of the same portrait, that is, it obtains the positions of each human keypoint in the training feature map (these positions correspond to the downsampled positions in the training image), and outputs the results representing the positions of the human keypoints. In one embodiment, the output of the second convolutional module is recorded as a human keypoint variable, where one human keypoint variable corresponds to all the human keypoints of a portrait. Currently, for the second regression module to perform regression, the human keypoint variable output by the second convolutional module should be the discretized position of the human keypoints. In this case, the human keypoint variable can be represented using 2*N*M bits of data. Here, M represents the pre-defined total number of human body keypoints that a person should have. For example, if a person is currently set to have 17 keypoints, then M represents 17. The 2*N*M bits of data include N-bit discretized data of the positions of M human body keypoints on the x-axis (this position can be represented by coordinates or distance from the corresponding anchor point) and N-bit discretized data of their positions on the y-axis. The x-axis and y-axis refer to the coordinate axes used for training the feature map, with their origin located at the upper left corner of the training feature map.

[0136] After the second convolutional module outputs the human keypoint variables, the second regression module can be used to regress these variables. To avoid negative distance values ​​after regression, the second regression module employs bidirectional distributed focal loss (DFL). Bidirectional DFL enables bidirectional prediction, meaning it can predict positive values ​​even when negative values ​​occur after regression, and it also improves the accuracy of human keypoint prediction. Currently, bidirectional distributed focal loss adds a bias coefficient and prediction granularity to the distributed focal loss. When using bidirectional distributed focal loss to regress the variable to be regressed, the bias coefficient is used to move the anchor point used in the regression process so that the moved anchor point is located at the center of the discrete bits of the variable to be regressed. In the second regression module, the variable to be regressed is the human keypoint variable. The bias coefficient is determined based on the discrete bits of the variable to be regressed, and the prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is the human keypoint variable, the result obtained after regression corresponds to the human keypoint training result. In other words, by setting hyperparameters (i.e., bias coefficients) to the DFL formula to move the anchor point position, the anchor point can be moved to the center of the discretized bits, thus ensuring that the distance after regression is positive, i.e., satisfying the condition of bidirectional prediction. By setting hyperparameters to the DFL formula to change the prediction granularity (i.e., prediction granularity), it is possible to predict distance (i.e., d) at a finer granularity (less than or equal to 1). t This allows for more accurate distance prediction, which can then be adapted to the prediction of key points on the human body.

[0137] In one embodiment, a bidirectional DFL can be represented by the following formula:

[0138]

[0139] Where N represents the number of discretized bits of the variable to be regressed, i represents the i-th bit in N bits, P(i) represents the i-th data point in the N-bit variable to be regressed, s represents the granularity of the prediction, (N-1) / 2 is the bias coefficient, and d t This represents the result after regression. Both N and s can be considered hyperparameters, and N, i, P(i), and d... t Related definitions and the terms N, i, P(i) and d in DFL t Same. Understandably, for the second regression module, the variable to be regressed is the human body keypoint variable.

[0140] In the above formula, (N-1) / 2 can be used to move the anchor point to the center of the discretized bits to ensure that the distance after regression is positive. s∈(0,1], the main purpose of s is to improve the prediction accuracy of bidirectional DFL regression.

[0141] Based on the above formula, the processing steps of two-way DFL include: multiplying each data point in the variable to be regressed by its corresponding digit to obtain the product result (i.e., iP(i)) for each data point; summing the product results and subtracting them from the bias coefficient; and then calculating the difference (i.e., ...) The regression result is obtained by multiplying the granularity hyperparameter by the regression parameter. Here, it can be understood that the variable to be regressed in the second regression module is the human body keypoint variable, which contains N discrete data points, and the sum of each data point equals 1 (i.e., ...). The bias coefficient is equal to (N-1) / 2, and the granularity hyperparameter is used to adjust the output result after regression (i.e., d). t () accuracy.

[0142] The second regression module uses bidirectional DFL to regress the human keypoint variables, obtaining the human keypoint training results, where one human keypoint variable corresponds to one human keypoint training result. Currently, when the second regression module uses the aforementioned bidirectional DFL formula to regress the human keypoint variables, d... t This is the result after regression. After the second regression module regresses the human body keypoint variables, 2*M d values ​​can be obtained. t And these represent the distances on the x-axis and y-axis of each human keypoint and anchor point (i.e., after (N-1) / 2 movements). The x-axis and y-axis refer to the coordinate axes used for training the feature map, with their origin located at the upper left corner of the training feature map. The resulting 2*M d... t This can be considered as the result of training a human keypoint. It can be understood that 2*M d-values ​​are obtained through bidirectional DFL. t Then, using the downsampling factor corresponding to the training feature map and 2*M d... t After multiplying them separately, we can obtain the distances on the x-axis and y-axis between each key point of the human figure in the training image and its corresponding anchor point, and thus obtain the coordinates of each key point, that is, predict the location where the key point of the human figure should be.

[0143] In one embodiment, the behavior prediction branch can be implemented using a convolutional network, that is, using a convolutional network to predict the behavior training results under each behavior category. When the behavior prediction branch predicts the behavior training results under each behavior category, it can also be considered as classifying behaviors; that is, the behavior prediction branch is used to perform the classification task. In one embodiment, the behavior prediction branch can be composed of 2×conv3+conv3+conv1. Currently, based on the features in the input training feature map (i.e., the features represented by each anchor point), the behavior prediction branch can obtain the behavior-related features of each portrait in the training image under each behavior category, and then determine the specific behavior of each portrait based on the obtained features, that is, obtain the behavior training result corresponding to each behavior category. In one embodiment, when determining the specific behavior of each portrait, the output behavior training result is the probability (i.e., confidence) of each behavior occurring under the corresponding behavior category. For example, if the behavior categories are sitting or standing, raising or not raising hands, and the specific head direction, then after processing the training feature map, the behavior prediction branch can obtain the probability of each portrait being sitting, standing, raising hands, not raising hands, and the probability of each specific head direction. At this point, the training results for the behavior category include the prediction score (i.e., probability) of the behavior prediction branch for each behavior under the corresponding behavior category. Therefore, when the behavior prediction branch performs behavior recognition, it can also be considered as performing binary classification for each specific behavior. Understandably, behavior classification based on convolutional networks is a relatively common technique, which will not be elaborated upon here.

[0144] It is understandable that human behaviors (such as sitting, standing, raising hands, not raising hands, head direction, etc.) are all related to human keypoints. That is, the location of each human keypoint will differ under different behaviors. Therefore, in one embodiment, features related to human keypoints can be used to strengthen the classification features (which are related to behavior) when the behavior prediction branch performs behavior classification. At this time, during the multi-task head network processing, when the behavior prediction branch in the task head network processes the corresponding training feature map, the keypoint features obtained by the human keypoint prediction branch in the same task head network based on the training feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the training feature map, so as to obtain the behavior training result corresponding to each behavior category based on the concatenated features. Specifically, during the human keypoint prediction branch processing, a convolutional network is needed to obtain the features of each human keypoint belonging to the same portrait in the training image (such as the features output in 2×conv3 in the second convolutional module). Currently, this feature is recorded as the keypoint feature, and then, the human keypoints of the same portrait can be predicted based on the keypoint feature. Currently, the keypoint features obtained by the human keypoint prediction branch are also input into the behavior prediction branch. During the behavior prediction branch processing, a convolutional network is also used to obtain behavior-related features (i.e., features used for classification) for each portrait in the training image under each behavior category. Currently, these features are denoted as behavior features. For example, the features output in 2×conv3 are the behavior features of each portrait under each behavior category. Then, the behavior prediction branch can concatenate the behavior features with keypoint features. Afterwards, the behavior prediction branch (e.g., conv3+conv1) uses the concatenated features to obtain the behavior training results corresponding to each behavior category. Optionally, the behavior features obtained by the behavior prediction branch do not help in the detection of human keypoints. Therefore, it is necessary to maintain the independence of the keypoint features, that is, the keypoint features should not be affected by behavior-related features. In the embodiment, when performing feature concatenation, the gradient backpropagation of behavior features to keypoint features can also be removed.

[0145] It should be noted that when the behavior prediction branch is also used to predict occluded human keypoints, the occlusion training results of human keypoints can also be obtained based on the aforementioned processing of the behavior prediction branch.

[0146] In one embodiment, when multiple prediction branches also include a human prediction branch, the human prediction branch can be implemented using a convolutional network, that is, using a convolutional network to predict whether a human body appears in each training feature map. When the human prediction branch predicts the human training result, it can also be considered as classifying the human body; that is, the human prediction branch is used to perform a classification task, specifically a binary classification task, that is, to classify human bodies and non-human bodies. In one embodiment, the human prediction branch can consist of 2×conv3+conv1. Currently, the human prediction branch, based on the features in the input training feature map (i.e., the features represented by each anchor point), can obtain human-related features in the training image, and then predict whether a human body appears, thus obtaining the human training result (which may include the confidence level of the human body). It is understood that behavior classification based on convolutional networks is a common technique, and will not be elaborated further here.

[0147] In one embodiment, when multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch can be implemented using a convolutional network. In this case, to reduce the error in predicting the head bounding box, a Direct-Fluid Imaging (DFL) can also be used for head bounding box prediction when using a convolutional network. Currently, because the head bounding box label and the body bounding box label share the same anchor point, the relative positional relationship between the head bounding box (such as the top-left and bottom-right corner points involved in regression) and the anchor point cannot be guaranteed. That is, the head bounding box can be located in any direction from the anchor point, for example... Figure 4 This is a schematic diagram of the head frame position provided in one embodiment of this application. The relative positions of the human body frame 31 and the head frame 32 assigned to anchor point a with respect to anchor point a are as follows: Figure 4 As shown, at this time, Figure 4 The coordinate system established with anchor point a is used to represent the distances between the human bounding box 31 and the head bounding box 32 and anchor point a. That is, the regression process can be assumed to have anchor point a as the origin, thus obtaining the distances between the human bounding box 31, the head bounding box 32, and the human keypoints and the origin. In the actual regression process, all positions and distances involved correspond to the positions and distances in the coordinate system used to train the feature maps. At this point, by... Figure 4 As can be seen, anchor point a is located inside the human body frame 31. Therefore, when using DFL to regress the human body frame variable, the obtained d t All are positive values, for example, Figure 4 d in p1x (That is, the distance between the top-left corner point p1 and the anchor point a along the x-axis in the coordinate system used for training the feature map) is equal to a x (i.e., the x-coordinate of anchor point a in the coordinate system used for training feature maps) minus p 1x The value of d (i.e., the coordinates of the top-left corner point p1 on the x-axis of the coordinate system used in the training feature map). p2x (That is, the distance between the bottom right corner point p2 and the anchor point a along the x-axis in the coordinate system used for training the feature map) is equal to p2x (i.e., the coordinates of the bottom right corner point p2 on the x-axis of the coordinate system used in the training feature map) minus a x The value of d p1x and d p2x All located at a x On one side, all values ​​are positive. However, anchor point a is located outside the head frame 32. In this case, when using DFL to predict the head frame, the regression result d... t There is a possibility of negative values, meaning that a positive distance cannot be obtained, for example, Figure 4 d in h1x (That is, the distance between the top-left corner point h1 and the anchor point a along the x-axis in the coordinate system used for training the feature map) is equal to a x (i.e., the coordinates of anchor point a on the x-axis of the coordinate system used in the training feature map) minus h 1x The value of d (i.e., the coordinates of the top-left corner point h1 on the x-axis of the coordinate system used in the training feature map). h2x (That is, the distance between the bottom right corner point h2 and the anchor point a along the x-axis in the coordinate system used for training the feature map) is equal to h 2x (i.e., the coordinates of the bottom right corner point h2 on the x-axis of the coordinate system used in the training feature map) minus a x (i.e., the coordinates of anchor point a on the x-axis of the coordinate system used for training the feature map) at this time, d h1x and d h2x Located at a x On both sides, i.e. d h2 The value is negative. Therefore, in one embodiment, when using DFL for head bounding box prediction, to avoid the possibility of negative values ​​in the regressed distance due to the head bounding box being located in any direction from the anchor point, a bidirectional DFL is used for regression. In this case, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each person image based on the input training feature map, and the third regression module is used to regress the head bounding box variables to obtain the head bounding box training result. The third regression module uses a bidirectional distributed focus loss.

[0148] The third convolutional module can also be considered a convolutional network. Its structure can be customized based on specific circumstances; for example, it might consist of 2×conv3+conv1. Currently, the third convolutional module obtains head-bound features in the training image based on the features in the input training feature map (i.e., the features represented by each anchor point). It then predicts the position of each portrait's head bounding box in the training feature map (this position corresponds to the downsampled position of the training image) based on these features, and outputs a result representing the head bounding box position. In one embodiment, the output of the second convolutional model is recorded as the head bounding box variable, with one head bounding box variable corresponding to one head bounding box. It can be understood that the processing of the third convolutional module is the same as that of the first convolutional module, the difference being that one predicts head bounding boxes and the other predicts body bounding boxes. Currently, the head bounding box variable also uses 4*N bits of data, similar to the body bounding box variable, and will not be elaborated further.

[0149] After the third convolution module outputs the head box variable, the third regression module can be used to regress the head box variable. To avoid obtaining a negative distance value after regression, the third regression module uses bidirectional DFL. Bidirectional DFL can achieve bidirectional prediction, that is, it can also predict to obtain a positive value even if a negative value appears after regression. Currently, the third regression module can also use formula (2) to implement bidirectional DFL. At this time, the variable to be regressed is the head box variable, and the result obtained after regression is the head box training result.

[0150] The third regression module uses bidirectional DFL to regress the head frame variable, obtaining the head frame training result, where each head frame variable corresponds to one head frame training result. Currently, when the third regression module uses formula (2) to regress the head frame variable, P(i) in the formula is the i-th data in the N-bit discretized data of the head frame variable, d t This is the result after regression. For example, when the position of the head frame is represented by the top left and bottom right corners, after regressing the head frame variable using the second regression model, four d values ​​can be obtained. t And respectively represent the distances on the x-axis and y-axis between the top left corner of the head frame and its corresponding anchor point (i.e., after moving (N-1) / 2), and the distances on the x-axis and y-axis between the bottom right corner of the head frame and its corresponding anchor point, and the four d values. t All values ​​are positive. The x-axis and y-axis refer to the coordinate axes used for training the feature map, with their origin located at the top left corner of the training feature map. For example, Figure 4 In the middle, d h2x After bidirectional DFL regression, we can obtain a x Subtract h 2x The value of (i.e., the coordinates of the lower right corner point h2 on the x-axis of the coordinate system used in the training feature map), i.e., d h2x It is a positive value. The four d values ​​obtained are...t This can be considered as a training result of a head bounding box. It can be understood that the downsampling factor corresponding to the training feature map and 4 d... t After multiplying these values ​​separately, we can obtain the distances on the x-axis and y-axis between the top-left corner of the head bounding box and its corresponding anchor point in the training image, as well as the distances on the x-axis and y-axis between the bottom-right corner of the head bounding box and its corresponding anchor point. This allows us to obtain the coordinates of the top-left and bottom-right corners of the head bounding box, thus predicting its location. It can be understood that if the position of the head bounding box is represented by its center point coordinates, width, and height, then the coordinates of the top-left and bottom-right corners of the head bounding box obtained from the third regression module can also yield the coordinates of its center point, width, and height.

[0151] As can be understood, by following the aforementioned processing method, all training results for the training image under each prediction branch can be obtained. The structures contained in each prediction branch are merely illustrative descriptions. In practical applications, each prediction branch can also contain other layers, such as pooling layers, to ensure that the corresponding training results can be output.

[0152] Step 150: Based on multiple training results and the real labels corresponding to each training result, construct multiple training losses for the corresponding task head network. The multiple training losses include human body bounding box loss, human body keypoint loss, and behavior loss corresponding to each behavior category.

[0153] In one embodiment, a supervised training method is used to train a multi-task detection model. In this case, the loss for each task can be constructed using the real labels and the corresponding training results. Currently, the loss constructed during the training process is denoted as the training loss, and there are multiple types of training losses. For example, when multiple prediction branches include a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch, the constructed training losses include the human bounding box loss, the human keypoint loss, and the behavior loss corresponding to each behavior category.

[0154] The human bounding box loss is a loss constructed based on the human bounding box labels and the training results. In one embodiment, the loss functions used to construct the human bounding box loss include DF Loss and CIOU Loss.

[0155] Here, DF Loss is the loss function corresponding to DFL. DF Loss transforms a regression problem into a classification problem, optimizing the probabilities of the two closest points to the corners of the human bounding box in a manner similar to cross-entropy. This allows the human bounding box prediction branch to quickly focus on the distribution of the neighborhood of the predicted corner points. The formula for DF Loss is as follows:

[0156] DFL(S i ,S i+1)=-((i+1-d t )log(S i )+(d t -i)log(S i+1 ),i≤d t ≤i+1 (3)

[0157] In the above formula, d t For d in the first regression module t i+1 and i are the features in the training feature map related to d. t The two closest positions, S i+1 and S i It represents the probability that two positions belong to the corresponding corner points. It is understood that DF Loss is an existing formula, and the substitution method when using DF Loss in the embodiments can refer to the substitution method in the prior art, which will not be described separately here.

[0158] CIOU Loss (Complete Intersection over Union Loss) is a loss function widely used in object detection tasks. Currently, the formula for using CIOU Loss when constructing the human bounding box loss adopts the existing CIOU Loss formula, which can be referred to in the existing CIOU Loss usage methods, and will not be described separately here.

[0159] Optionally, a loss value can be obtained based on DFLoss and another based on CIOU Loss. The human body bounding box loss can be obtained by adding the two loss values. Alternatively, different weights can be assigned to the two loss values, and the human body bounding box loss can be obtained by weighting them.

[0160] Understandably, each task head network can obtain a human bounding box loss. In the calculation of the human bounding box loss of each task head network, the human bounding box labels used are obtained by downsampling the corresponding training feature map by a downsampling factor.

[0161] Human keypoint loss is a loss function constructed based on human keypoint labels and training results. In one embodiment, the loss functions used to construct the human keypoint loss include DF Loss and OKS Loss.

[0162] The description of DF Loss can be found in the section on DF Loss used to construct the human bounding box loss, as both are used in the same way and will not be described again here. OKS Loss (Object Keypoint Similarity Loss) measures the similarity between predicted keypoints (i.e., human keypoint results) and ground truth keypoints (i.e., human keypoint labels). Currently, the formula for OKS Loss used when constructing the human keypoint loss follows the existing OKS Loss formula, and its usage can be found in the existing OKS Loss section; it will not be described again here.

[0163] Optionally, a loss value can be obtained based on DF Loss, and another loss value can be obtained based on OKS Loss. The human keypoint loss can be obtained by adding the two loss values. Alternatively, different weights can be set for the two loss values, and the human keypoint loss can be obtained by weighting them.

[0164] Understandably, each task head network can obtain a human keypoint loss. In the calculation of the human bounding box loss of each task head network, the human keypoint labels used are obtained by downsampling according to the downsampling factor of the corresponding training feature map.

[0165] In one embodiment, the behavior loss corresponds to each behavior category. The behavior loss is constructed based on the behavior labels under the corresponding behavior category and the training results of the behavior under that behavior category. In one embodiment, the loss function used to construct the behavior loss is a two-weighted loss function.

[0166] The dual-weight loss function considers two aspects of weights. One is the weight of the specific behavior category within the behavior category, which can be denoted as the category weight. Each behavior within each category has a corresponding category weight, and the specific value of the category weight can be set according to the actual situation. For example, in a teaching scenario, under the behavior categories of raising and not raising hands, there are two behavior classifications: raising hands and not raising hands. That is, there are two behaviors within this behavior category: raising hands and not raising hands. Furthermore, the number of images of people not raising hands in each training image is generally much greater than the number of images of people raising hands (the difference can reach more than 50 times). Therefore, a category weight can be set for raising hands and another for not raising hands, with the category weight for not raising hands being less than that for raising hands, to address the problem of the imbalance in the number of different behaviors within the same behavior category. The other is the sample weight, which can also be understood as the matching score when assigning positive samples. It can be understood that during the training of the multi-task detection model, it is necessary to assign positive samples to the anchor points on the training feature map according to the real labels, that is, to determine which anchor points are positive samples and their correspondence with the real labels. For example, the SimOTA method is used to assign each real label to its corresponding anchor point. During the assignment process, all real labels of a portrait can correspond to multiple anchor points (i.e., real labels correspond to multiple positive sample anchor points). This generates a matching score between the positive sample anchor points and their corresponding real labels (this matching score is the existing technical method). The higher the matching score, the more representative the anchor point's features are of the corresponding real label. The matching score reflects the importance of assigning anchor points; therefore, it should also be considered when calculating the behavioral loss. In one embodiment, a dual-weight loss function, denoted as DWCE Loss, is designed based on class weights and matching scores. This is an improvement on CE Loss. The loss formula for the behavioral loss is as follows:

[0167]

[0168] Where DWCELoss represents the behavioral loss (i.e., the specific value calculated), i represents the i-th anchor point in the training feature map, and l i p represents the behavior label corresponding to the i-th anchor point (i.e., the matched behavior label). i W represents the prediction score (i.e., the confidence level of the behavior training result) obtained when the behavior prediction branch performs behavior prediction. c This represents the category weight corresponding to the behavior label. For example, when the behavior training result is "raising hand", W... c W represents the category weight corresponding to raising hands. p This represents the matching score between the i-th anchor and its corresponding behavior label.

[0169] It should be noted that during the training process, when the behavior prediction branch makes predictions, the behavior training results corresponding to the behavior category include the prediction score of the behavior prediction branch for each behavior under the corresponding behavior category. Specifically, each anchor point corresponding to the same portrait in the training feature map has a corresponding prediction score for each behavior. That is, the behavior prediction branch can obtain the prediction score for each behavior of a feature attribute based on the feature represented by an anchor point. Based on formula (4), the aforementioned process of determining the behavior loss may include: obtaining the natural logarithm loss (i.e., l) corresponding to each prediction score based on each prediction score under the behavior category and the corresponding behavior label. i log(p i The natural log loss corresponding to each predicted score is multiplied by the category weight corresponding to the corresponding behavior label to obtain the first multiplication result (i.e., W) for each predicted score. c l i log(p i Each behavior within a behavior category has a corresponding category weight. The more behavior labels corresponding to a behavior appear in each training image, the smaller the corresponding category weight. The first multiplication results corresponding to each predicted score under the same anchor point (i.e., the predicted scores of each behavior under the aforementioned behavior category obtained based on the same anchor point) are summed (i.e., ∑...). c W c l i log(p i After that, multiply it by the matching score corresponding to the anchor point to obtain the second multiplication result corresponding to the anchor point (i.e., W). p (∑ c W c l i log(p i Matching score refers to the matching score between the anchor point and the behavior label under the behavior category; after adding the second multiplication result of each anchor point corresponding to the same portrait, divide the sum of the matching scores of each anchor point corresponding to the portrait to obtain the behavior loss of the portrait under the behavior category.

[0170] Understandably, the corresponding behavioral loss can be obtained based on formula (4) for each behavioral category. For example, sitting or standing corresponds to a behavioral loss, raising or not raising a hand corresponds to a behavioral loss, and the specific head direction corresponds to a behavioral loss.

[0171] At this point, the various training losses corresponding to each task head network include human bounding box loss, human keypoint loss, and behavior loss for each behavior category.

[0172] In one embodiment, when the behavior prediction branch is also used to predict occluded human keypoints, the various training losses corresponding to each task head network should also include a human keypoint occlusion loss. The human keypoint occlusion loss is a loss constructed based on the human keypoint occlusion labels and the human keypoint occlusion training results. In one embodiment, the loss function used to construct the human keypoint occlusion loss can be the Binary Cross Entropy Loss (BCELoss). BCE Loss is used in binary classification problems to measure the difference between the model output (currently the human keypoint occlusion training result) and the true label (currently the human keypoint occlusion label). Currently, the formula for using BCE Loss when constructing the human keypoint occlusion loss adopts the existing BCE Loss formula, and the usage of existing BCE Loss methods can be referred to; it will not be described separately here.

[0173] In one embodiment, when multiple prediction branches also include a human prediction branch, the various training losses corresponding to each task head network should also include a human loss. The human loss is a loss constructed based on human labels and human training results. In one embodiment, the loss function used to construct the human loss is BCE Loss. Currently, the formula for using BCE Loss to construct the human loss can adopt the existing BCE Loss formula; refer to the existing BCE Loss usage methods, which will not be described further here.

[0174] In one embodiment, when multiple prediction branches also include a head bounding box prediction branch, the various training losses corresponding to each task head network should also include a head bounding box loss. The head bounding box loss is a loss constructed based on the head bounding box labels and the head bounding box training results. In one embodiment, the loss functions used to construct the head bounding box loss include DF Loss and CIOU Loss. The process of constructing the head bounding box loss can refer to the process of constructing the human bounding box loss function, and will not be elaborated further here.

[0175] Step 160: Update the model parameters of the multi-task detection model based on the various training losses corresponding to each task head network, so as to train the multi-task detection model.

[0176] For example, after the aforementioned steps, each task head network has multiple corresponding training losses. Then, the model parameters used by the multi-task detection model can be adjusted based on these training losses. Optionally, the same training losses for each task head network can be summed first to obtain the total loss for that task. Then, the total losses for each task can be summed or weighted to obtain the total loss of the multi-task detection model. Finally, the model parameters used by the multi-task detection model are updated based on the total loss. It is understood that adjusting model parameters based on the loss value determined by the loss function is an already implemented technique and will not be described further here.

[0177] Following the aforementioned training method, after training the multi-task detection model multiple times (i.e., repeatedly inputting training images, obtaining training loss, and adjusting model parameters), the multi-task detection model tends to stabilize. At this point, the training of the multi-task detection model can be considered complete. The stabilization of the multi-task detection model means that the training results output by the model are basically stable (there will be no significant differences in results obtained when processing the same image multiple times) and the accuracy meets the requirements.

[0178] After training the multi-task detection model, it can be applied to perform multi-task detection.

[0179] The training method for the aforementioned multi-task detection model is described below as an example, wherein... Figure 5 This is a schematic diagram of a multi-task detection model provided in one embodiment of this application, with reference to... Figure 5 This multi-task detection model consists of a Backbone network 41, a Neck network 42, and a head network 43. The head network contains three task head networks, denoted as task head network 431, task head network 432, and task head network 433. Each task head network has the same structure; therefore... Figure 5 The example illustrates a specific structure of a task head network 432, see reference. Figure 5 The task head network 432, from top to bottom, includes a human body prediction branch (2×conv3+conv1), a human body bounding box prediction branch (2×conv3+conv1 and DFL), a head bounding box prediction branch (2×conv3+conv1 and bidirectional DFL), a human keypoint prediction branch (2×conv3+conv3+conv1 and bidirectional DFL), and a behavior prediction branch (2×conv3+conv3+conv1). The behavior prediction branch can predict whether the person is standing or sitting, whether their hand is raised or not, head direction, and whether human keypoints are hidden. It should be noted that... Figure 5In this diagram, each circle ("○") represents a category. For example, the human detection branch only needs to identify whether something is a human body, which is one category, so it has one circle ("○"). In the behavior prediction branch, identifying standing or sitting requires classifying two behaviors (standing and sitting), so there are two circles ("○"). In practice, an additional category of "other" can be added to the standing and sitting categories, resulting in three circles ("○"). Identifying raised or unraised hands also requires classifying two behaviors (raised and unraised hands), resulting in two circles ("○"). In practice, an additional category of "other" can be added to the raised and unraised hand categories, resulting in three circles ("○"). Identifying head direction requires classifying five behaviors (head up, head down, head tilted, head back, and others), resulting in five circles ("○"). Identifying whether key points on the human body are hidden only requires classifying one situation (hidden situation), so there is one circle ("○").

[0180] In this multi-task detection model, training images with ground truth labels are input and processed by the Backbone and Neck networks to obtain three training feature maps of different scales. Each training feature map is then input into a corresponding task head network. The human prediction branch in the task head network outputs the human training result, `body_score` (the probability of a human body). Based on the human training result and the corresponding ground truth label, the human loss is obtained through BCE Loss. The human bounding box prediction branch in the task head network outputs the human bounding box training result, `body_box` (the position-related representation of the human bounding box). Based on the human bounding box training result and the corresponding ground truth label, the human bounding box loss is obtained through DF Loss and CIOU Loss. The head bounding box prediction branch in the task head network outputs the head bounding box training result, `head_box` (the position-related representation of the head bounding box). Based on the head bounding box training result and the corresponding ground truth label, the head bounding box loss is obtained through DF Loss and CIOU Loss. The human keypoint prediction branch in the task head network outputs the training results of human keypoints, i.e., keypoints (which are position-related representations of human keypoints). Based on the training results of human keypoints and the corresponding ground truth labels, the human keypoint loss can be obtained through DF Loss and OKS Loss. The behavior prediction branch in the task head network outputs the training results of standing or sitting behavior, raising or not raising hands behavior, specific head orientation, and whether human keypoints are hidden, i.e., pose_score (which represents the confidence of recognizing the behavior, such as the confidence of recognizing sitting and standing), hand_score (which represents the confidence of recognizing the behavior, see pose_score), head_score (which represents the confidence of recognizing the behavior, see pose_score), and kpt_vis_score (which represents the confidence of recognizing that human keypoints are hidden). The keypoint features obtained in the human keypoint prediction branch are concatenated with the behavior features obtained in the behavior prediction branch, and gradient backpropagation is removed during concatenation. Based on the training results of standing or sitting behavior and the corresponding ground truth labels, the corresponding behavioral loss can be obtained through DWCE Loss. Based on the training results of raising or not raising hands behavior and the corresponding ground truth labels, the corresponding behavioral loss can be obtained through DWCE Loss. Based on the specific head direction and the corresponding ground truth labels, the corresponding behavioral loss can be obtained through DWCE Loss. Based on the training results of whether human keypoints are hidden and the corresponding ground truth labels, the corresponding human keypoint occlusion behavior loss can be obtained through BCE Loss.Subsequently, the model parameters of the multi-task detection model are updated based on the human body loss, human body bounding box loss, human head bounding box loss, human body keypoint loss, three types of behavior loss, and human body keypoint occlusion loss of each task head network, so as to complete the training of the multi-task detection model.

[0181] The above describes a multi-task detection model designed to perform at least human bounding box detection, human keypoint detection, and behavior detection. Furthermore, by inputting training images into this model, multi-stage detection is unnecessary; that is, a single-person image is not required before other detections. Multiple tasks can be performed in a single stage (using one model), avoiding the long inference time associated with multi-stage methods. Moreover, the multi-task detection model has a smaller footprint compared to the multiple models used in multi-stage methods. Additionally, since it directly predicts human bounding boxes, human keypoints, and behaviors, end-to-end training is achieved, avoiding object ambiguity and severe coupling between models. Furthermore, during task head network processing, training results predicted based on the same anchor point correspond to the same human image, eliminating the need for matching training results and improving the processing efficiency of the multi-task detection model. Furthermore, the prediction branches in the task head network can be added or removed based on actual conditions without modifying other structures of the multi-task detection model. For example, human body prediction branches and head bounding box prediction branches can be added to achieve human body detection and head bounding box detection, enriching the types of tasks the multi-task detection model can handle and broadening its application scenarios. During training, all ground truth labels for the same image are assigned to the same anchor point, with each anchor point corresponding to only one image's ground truth labels. This shared anchor point mechanism enables end-to-end training, ensuring that all training results obtained by the task head network based on the same anchor point correspond to the same image, improving the accuracy and processing efficiency of the multi-task detection model. In behavior prediction, considering the uneven distribution of different behaviors within the same behavior category and the varying matching degrees between anchor points and ground truth labels, a behavior loss is designed. This involves adding category weights and sample weights to the loss function to classify behaviors within each behavior category, improving the accuracy of the multi-task detection model. Additionally, the keypoint features obtained from the human keypoint prediction branch can be used by the behavior prediction branch, further improving the accuracy of behavior prediction. Furthermore, a regression module has been added for predicting human bounding boxes, head bounding boxes, and human keypoints. This module treats the predicted target locations as a distribution and performs regression analysis, improving the flexibility and accuracy of the predictions. Additionally, a two-way DFL (Distributed Flow Algorithm) has been designed. By setting a bias coefficient, the regression results for the head bounding box and human keypoint variables are guaranteed to be positive, facilitating subsequent processing. Moreover, adjusting the granularity of the predictions further enhances the accuracy of the regression results.

[0182] Understandably, once a multi-task detection model has been trained, it can be deployed for application. Therefore, one embodiment of this application also provides a multi-task detection method. This method uses the aforementioned multi-task detection model to respectively implement tasks such as human bounding box detection, human keypoint detection, and behavior detection. Furthermore, the detection process does not need to be divided into multiple stages; that is, it does not require obtaining a single-person image before detection. This avoids problems such as large model overhead and long inference time, object ambiguity, and severe coupling between models during the detection process, resulting in higher detection accuracy and efficiency.

[0183] The multi-task detection method provided in this application embodiment can be executed by a multi-task detection device. This multi-task detection device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Currently, the multi-task detection device can be a computer, server, or other device with data processing capabilities and the ability to deploy neural networks (such as multi-task detection models). The multi-task detection device and the training device for the multi-task detection model can be the same entity or different entities; this embodiment does not limit this.

[0184] A multi-task detection device includes at least a processor and memory. The processor and memory can be connected via a bus or other means.

[0185] The number of processors can be one or more. Processors may include processing units such as AP, GPU, and CPU.

[0186] Memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-task detection device in the embodiments of this application. Memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the multi-task detection device, etc. Furthermore, memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the multi-task detection device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] In addition, the multi-task detection device may also include one or more components such as a display screen, a component for accessing a network (such as the Internet), a communication interface, a power supply, a speaker, a camera, and an input device, etc., but the embodiments are not limited thereto.

[0188] Based on the aforementioned hardware architecture, the multi-task detection device is equipped with at least one operating system. This operating system can be Android, Windows, or Linux, among others. Using this installed operating system, the device can train a multi-task detection model to implement a multi-task detection method.

[0189] Figure 6 A flowchart of a multi-task detection method provided in one embodiment of this application is shown below. Figure 6 The multi-task detection method includes steps 510-550:

[0190] Step 510: Obtain the image to be detected.

[0191] The image to be detected is the image that needs to be processed during the application of the multi-task detection model. For example, in a teaching scenario, the image to be detected is a classroom image, that is, an image of students listening to a lesson. The source of the image to be detected is not currently limited; for example, a school can obtain a classroom image by taking pictures of students listening to a lesson in the classroom.

[0192] It is understandable that the image to be detected does not contain any real labels.

[0193] Step 520: Input the image to be detected into the multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The behavior prediction branch is used to predict behaviors under multiple behavior categories.

[0194] For example, after acquiring the image to be detected, the image is input into a multi-task detection model. The description of the multi-task detection model can be found in the aforementioned training method section. It is understood that the multiple prediction branches include at least one of a human body prediction branch and a head bounding box prediction branch, and the behavior prediction branch can also be used to predict occluded human body keypoints.

[0195] Step 530: The feature extraction network obtains multiple feature maps of different scales from the image to be detected, and each feature map is input into the corresponding task head network. One task head network corresponds to one feature map.

[0196] During application, the feature extraction network processes the image to be detected to extract multiple feature maps of different scales, which are then input into the corresponding task head network. This process can be referred to in the relevant description of step 130, the difference being that one extracts feature maps and the other extracts training feature maps.

[0197] Step 540: Each prediction branch in the task head network processes the corresponding feature map to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human key point prediction results, and behavior prediction results corresponding to each behavior category. During the task head network processing, the various prediction results obtained based on the same anchor point in the feature map correspond to the same human figure.

[0198] In one embodiment, during application, each detection result output by each prediction branch in the task head network after predicting the feature map is recorded as a prediction result. That is, the human bounding box prediction branch outputs the human bounding box prediction result, the human keypoint prediction branch outputs the human keypoint prediction result, and the behavior prediction branch outputs the behavior prediction result for each behavior category. It is understood that when multiple prediction branches also include at least one of the human bounding box prediction branch and the head bounding box prediction branch, multiple prediction results also correspond to at least one of the human bounding box prediction result and the head bounding box prediction result. Furthermore, when the behavior prediction branch is also used to predict occluded human keypoints, multiple prediction results also include human keypoint occlusion prediction results. The descriptions of the human bounding box prediction result, human keypoint prediction result, behavior prediction results for each behavior category, human prediction result, head bounding box prediction result, and human keypoint occlusion prediction result can refer to the descriptions of the human bounding box training result, human keypoint training result, behavior training result for each behavior category, human training result, head bounding box training result, and human keypoint occlusion training result in the aforementioned embodiments. The difference is that one is applied during the application process, and the other is applied during the training process.

[0199] For example, when the behavior prediction branch in the task head network processes the corresponding feature map, the keypoint features obtained by the human keypoint prediction branch in the same task head network based on the feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the feature map. The behavior prediction result for each behavior category is then obtained based on the concatenated features. This process is similar to the process in which the behavior prediction branch in the task head network processes the corresponding training feature map, concatenating the keypoint features obtained by the human keypoint prediction branch in the same task head network based on the training feature map with the behavior features obtained by the behavior prediction branch based on the training feature map, and obtaining the behavior training result for each behavior category based on the concatenated features. This will not be elaborated further here.

[0200] In one embodiment, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input feature map, and the first regression module is used to regress the human bounding box variables to obtain the human bounding box prediction result. The first regression module uses distributed focus loss. The human keypoint prediction branch consists of a second convolutional module and a second regression module. The second convolutional module is used to predict the human keypoint variables corresponding to each human image based on the input feature map, and the second regression module is used to regress the human keypoint variables to obtain the human keypoint prediction result. The second regression module uses bidirectional distributed focus loss. When multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each human image based on the input feature map, and the third regression module is used to regress the head bounding box variables to obtain the head bounding box prediction result. The third regression module uses bidirectional distributed focus loss.

[0201] The bidirectional distributed focus loss adds a bias coefficient and prediction granularity to the aforementioned distributed focus loss. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization bits of the variable to be regressed. The variable to be regressed in the second regression module is the human keypoint variable, and the variable to be regressed in the third regression module is the head frame variable. The bias coefficient is determined according to the discretization bits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is the human keypoint variable, the result corresponds to the human keypoint prediction result; when the variable to be regressed is the head frame variable, the result corresponds to the head frame prediction result.

[0202] The processing procedures of the first convolution module, the first regression module, the second convolution module, the second regression module, the third convolution module, and the third regression module, as well as the explanation of the bidirectional DFL, can be found in the relevant descriptions in the foregoing embodiments.

[0203] It is understood that the processing procedures for the behavior prediction branch, the human body prediction branch, and the head bounding box prediction branch can also refer to the relevant descriptions in the foregoing embodiments.

[0204] Step 550: Obtain the final detection result of the corresponding human image in the image to be detected based on the prediction results of the same human image output by each task head network. Each prediction result corresponds to a final detection result.

[0205] Understandably, since each task head network outputs multiple prediction results for each portrait, multiple prediction results will be obtained for the same task. For example, in the task of human bounding box prediction, each task head network will output a human bounding box prediction result. At this point, it is also necessary to obtain the final detection result for that portrait under that task based on the prediction results for the same portrait output by each task head network under the same task. For example, the final human bounding box detection result for that portrait can be obtained based on the human bounding box prediction results for the same portrait output by each task head network. At this point, each detectable portrait under each task has a corresponding final detection result. Currently, the final detection result for each portrait includes the detection result of the human bounding box, the detection result of human keypoints, and the behavior detection result for each behavior category. It can also include the human detection result, the head bounding box detection result, and the human keypoint occlusion detection result.

[0206] In one embodiment, the final detection result for the same task can be obtained by deduplication based on multiple prediction results for the same task. The deduplication method used is not currently limited; for example, Soft-NMS can be used. Non-Maximum Suppression (NMS) is a commonly used technique in computer vision to eliminate redundant bounding boxes in object detection tasks. Soft-NMS is an improvement on NMS. Soft-NMS considers both overlap and score during NMS, addressing the problems NMS faces when dealing with dense and small objects. When two bounding boxes (corresponding to two prediction results for the same task) overlap, Soft-NMS does not directly discard the bounding box with the lower score, but rather reduces its score and considers both the scores and overlap when calculating the final score. This way, even if two bounding boxes highly overlap, both may be retained, thus improving the detection performance for dense and small objects. It is understood that in practical applications, other methods can also be used to obtain the final detection result for the same task based on multiple prediction results.

[0207] The final detection results obtained can be considered as the output of the multi-task detection model.

[0208] The above describes a method that, by inputting the image to be detected into a multi-task detection model, the model's feature extraction network obtains multiple feature maps at different scales. Each task head network of the multi-task detection model can then process one feature map, and each prediction branch in the task head network can obtain a corresponding prediction result based on the feature map, thus achieving multi-task detection. Currently, the prediction results include at least human bounding box prediction results, human keypoint prediction results, and behavior prediction results for each behavior category. Then, the final detection result for each task is obtained based on the multiple prediction results corresponding to each task. This technique solves the technical problem of long inference time when using multi-stage detection methods in related technologies. Furthermore, compared to the models used in multi-stage methods, the multi-task detection model has a smaller footprint. Moreover, since directly predicting human bounding boxes, human keypoints, and behaviors avoids object ambiguity and severe coupling between models, and during the task head network processing, prediction results based on the same anchor point correspond to the same human image, eliminating the need for prediction result matching and improving the processing efficiency of the multi-task detection model. Furthermore, the prediction branches in the task head network can be added or removed based on actual conditions without modifying other structures of the multi-task detection model. For example, human body prediction branches and head bounding box prediction branches can be added to achieve human body detection and head bounding box detection, enriching the types of tasks the multi-task detection model can handle and broadening its application scenarios. Moreover, the keypoint features obtained from the human keypoint prediction branch can be used by the behavior prediction branch, further improving the accuracy of behavior prediction. Additionally, a regression module is added for human bounding boxes, head bounding boxes, and human keypoint prediction, treating the predicted target location as a distribution and performing regression, improving the flexibility and accuracy of prediction. Furthermore, a bidirectional DFL is designed, where setting a bias coefficient ensures that the regression results for head bounding box variables or human keypoint variables are positive, facilitating subsequent processing. Setting the granularity of prediction can improve the accuracy of the regression results, meeting the requirement of regressing human bounding boxes, head bounding boxes, and human keypoints from the same anchor point.

[0209] One embodiment of this application also provides a multi-task detection device. Figure 7 This is a schematic diagram of a multi-task detection device provided in one embodiment of this application, with reference to... Figure 7 The multi-task detection device includes a first acquisition unit 601, a first input unit 602, a first feature extraction unit 603, a first prediction unit 604, and a result determination unit 605.

[0210] The system includes a first acquisition unit 601 for acquiring an image to be detected; a first input unit 602 for inputting the image to be detected into a multi-task detection model, the multi-task detection model including a feature extraction network and multiple task head networks, each task head network including multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch and a behavior prediction branch, the behavior prediction branch being used to predict behaviors under multiple behavior categories; and a first feature extraction unit 603 for obtaining multiple feature maps of different scales from the image to be detected by the feature extraction network and inputting each feature map into the corresponding task head network, one task head network corresponding to one feature map. The first prediction unit 604 is used to process the corresponding feature maps by each prediction branch in the task head network to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human keypoint prediction results, and behavior prediction results corresponding to each behavior category. The various prediction results obtained by the task head network based on the same anchor point in the feature map during the processing correspond to the same human figure. The result determination unit 605 is used to obtain the final detection result of the corresponding human figure in the image to be detected based on the prediction results of the same human figure output by each task head network. Each prediction result corresponds to one final detection result.

[0211] In one embodiment of this application, the plurality of prediction branches further include at least one of a human body prediction branch and a head frame prediction branch. Correspondingly, the plurality of prediction results also include at least one of a human body prediction result and a head frame prediction result. The behavior prediction branch is also used to predict occluded human body key points. Correspondingly, the plurality of prediction results also include human body key point occlusion prediction results.

[0212] In one embodiment of this application, it further includes: a first splicing unit, used to splice the key point features obtained by the human key point prediction branch in the same task head network based on the feature map and the behavior features obtained by the behavior prediction branch based on the feature map when the behavior prediction branch in the task head network processes the corresponding feature map, so as to obtain the behavior prediction result corresponding to each behavior category based on the spliced ​​features.

[0213] In one embodiment of this application, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict human bounding box variables corresponding to each human image based on the input feature map, and the first regression module is used to regress the human bounding box variables to obtain the human bounding box prediction result. The first regression module uses distributed focus loss. The human keypoint prediction branch consists of a second convolutional module and a second regression module. The second convolutional module is used to predict human keypoint variables corresponding to each human image based on the input feature map, and the second regression module is used to regress the human keypoint variables to obtain the human keypoint prediction result. The second regression module uses bidirectional distributed focus loss. When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict head bounding box variables corresponding to each human image based on the input feature map, and the third regression module is used to regress the head bounding box variables to obtain the head bounding box prediction result. The third regression module uses bidirectional distributed focus loss.

[0214] In one embodiment of this application, the bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization digits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is a human keypoint variable, and the variable to be regressed corresponding to the third regression module is a head frame variable. The bias coefficient is determined according to the discretization digits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is a human keypoint variable, the result corresponds to a human keypoint prediction result. When the variable to be regressed is a head frame variable, the result corresponds to a head frame prediction result.

[0215] The multi-task detection device provided in this application embodiment is included in the multi-task detection equipment and can be used to execute the multi-task detection method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0216] One embodiment of this application also provides a training apparatus for a multi-task detection model. Figure 8 This is a schematic diagram of the structure of a training device for a multi-task detection model provided in one embodiment of this application, with reference to... Figure 8 The multi-task detection device includes a second acquisition unit 701, a second input unit 702, a second feature extraction unit 703, a second prediction unit 704, a loss construction unit 705, and a parameter update unit 706.

[0217] The second acquisition unit 701 is used to acquire a training dataset, which contains multiple training images. Each portrait in each training image has multiple corresponding ground truth labels, including human bounding box labels, human keypoint labels, and behavior labels under multiple behavior categories. The second input unit 702 is used to input the training images into a multi-task detection model, which includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including human bounding box prediction branches, human keypoint prediction branches, and behavior prediction branches. The second feature extraction unit 703 is used to obtain multiple training feature maps of different scales from the training images by the feature extraction network and input each training feature map into the corresponding task head network, with one task head network corresponding to one training feature map. The second prediction unit 704 uses... Each prediction branch in the task head network processes the corresponding training feature map to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. The various training results predicted based on the same anchor point in the training feature map during the task head network processing correspond to the same human image. The loss construction unit 705 is used to construct multiple training losses corresponding to the task head network based on the multiple training results and the real labels corresponding to each training result. The multiple training losses include human bounding box loss, human keypoint loss, and behavior loss corresponding to each behavior category. The parameter update unit 706 is used to update the model parameters of the multi-task detection model based on the multiple training losses corresponding to each task head network to train the multi-task detection model.

[0218] In one embodiment of this application, the plurality of prediction branches further includes at least one of a human prediction branch and a head bounding box prediction branch. Correspondingly, the plurality of ground truth labels also include at least one of a human body label and a head bounding box label. The plurality of training results also include at least one of a human body training result and a head bounding box training result. The plurality of training losses also include at least one of a human body loss and a head bounding box loss. The behavior prediction branch is also used to predict occluded human keypoints. Correspondingly, the plurality of ground truth labels also include human keypoint occlusion labels. The plurality of training results also include human keypoint occlusion training results. The plurality of training losses also include human keypoint occlusion losses.

[0219] In one embodiment of this application, multiple ground truth labels corresponding to the same portrait in the training image are assigned to the same anchor point in the training feature map, and each anchor point is assigned multiple ground truth labels for only one portrait.

[0220] In one embodiment of this application, the behavior training result corresponding to the behavior category includes the prediction score of the behavior prediction branch for each behavior under the corresponding behavior category. Each anchor point corresponding to the same portrait in the training feature map has a corresponding prediction score for each behavior. Accordingly, the process of determining the behavior loss corresponding to a behavior category includes: obtaining the natural logarithm loss corresponding to each prediction score based on each prediction score and the corresponding behavior label under the behavior category; multiplying the natural logarithm loss corresponding to each prediction score by the category weight corresponding to the corresponding behavior label to obtain the first [predicted loss] corresponding to each prediction score. The multiplication results show that each behavior under the behavior category has a corresponding category weight. The more behavior labels corresponding to the behavior appear in each training image, the smaller the corresponding category weight. The first multiplication results corresponding to each prediction score under the same anchor point are added together and multiplied by the matching score corresponding to the anchor point to obtain the second multiplication result corresponding to the anchor point. The matching score refers to the matching score between the anchor point and the behavior label under the behavior category. The second multiplication results corresponding to each anchor point of the same portrait are added together and divided by the sum of the matching scores of each anchor point corresponding to the portrait to obtain the behavior loss of the portrait under the behavior category.

[0221] In one embodiment of this application, it further includes: a second splicing unit, used to splice the key point features obtained by the human key point prediction branch in the same task head network based on the training feature map and the behavior features obtained by the behavior prediction branch based on the training feature map when the behavior prediction branch in the task head network processes the corresponding training feature map, so as to obtain the behavior training result corresponding to each behavior category based on the spliced ​​features.

[0222] In one embodiment of this application, the human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict human bounding box variables corresponding to each human image based on the input training feature map, and the first regression module is used to regress the human bounding box variables to obtain human bounding box training results. The first regression module uses distributed focus loss. The human keypoint prediction branch consists of a second convolutional module and a second regression module. The second convolutional module is used to predict human keypoint variables corresponding to each human image based on the input training feature map, and the second regression module is used to regress the human keypoint variables to obtain human keypoint training results. The second regression module uses bidirectional distributed focus loss. When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict head bounding box variables corresponding to each human image based on the input training feature map, and the third regression module is used to regress the head bounding box variables to obtain head bounding box training results. The third regression module uses bidirectional distributed focus loss.

[0223] In one embodiment of this application, the bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization bits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is a human keypoint variable, and the variable to be regressed corresponding to the third regression module is a head frame variable. The bias coefficient is determined according to the discretization bits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is a human keypoint variable, the result corresponds to the human keypoint training result. When the variable to be regressed is a head frame variable, the result corresponds to the head frame training result.

[0224] The training device for the multi-task detection model provided in this application embodiment is included in the training equipment for the multi-task detection model and can be used to execute the training method for the multi-task detection model provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0225] It is worth noting that in the embodiments of the above devices, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of this application.

[0226] One embodiment of this application also provides a multi-task detection device. Figure 9 This is a schematic diagram of a multi-task detection device provided in one embodiment of this application. (Reference) Figure 9 The multi-task detection device includes a processor 81 and a memory 82. The processor 81 and memory 82 can be connected via a bus or other means. The memory 82 stores one or more programs; when one or more programs are executed by one or more processors 81, the processors 81 implement the multi-task detection method described in any of the foregoing embodiments. The relevant details of each component can be found in the foregoing description.

[0227] The aforementioned multi-task detection device is used to execute any multi-task detection method and has corresponding functions and beneficial effects. For specific details not described here, please refer to the relevant descriptions of the aforementioned multi-task detection methods.

[0228] One embodiment of this application also provides a training device for a multi-task detection model. Figure 10 This is a schematic diagram of the structure of a training device for a multi-task detection model provided in one embodiment of this application. (Reference) Figure 10 The training device for this multi-task detection model includes a processor 91 and a memory 92. The processor 91 and the memory 92 can be connected via a bus or other means. The memory 92 stores one or more programs; when one or more programs are executed by one or more processors 91, the processors 91 implement the training method for the multi-task detection model described in any of the foregoing embodiments. For details on each component, please refer to the foregoing description.

[0229] The training device for the aforementioned multi-task detection model is used to execute the training method for any multi-task detection model, and has corresponding functions and beneficial effects. For specific details not described here, please refer to the relevant description of the training method for the aforementioned multi-task detection model.

[0230] One embodiment of this application also provides a storage medium containing computer-executable instructions, which, when executed by a processor, are used to perform relevant operations in the multi-task detection method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0231] One embodiment of this application also provides a storage medium containing computer-executable instructions, which, when executed by a processor, are used to perform relevant operations in the training method of the multi-task detection model provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0232] Those skilled in the art will understand that embodiments of this application may provide methods, systems, or computer program products.

[0233] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing module of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing module of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0234] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0235] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0236] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A multi-task detection method, characterized in that, include: Acquire the image to be detected; The image to be detected is input into a multi-task detection model, which includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The behavior prediction branch is used to predict behaviors under multiple behavior categories. The feature extraction network obtains multiple feature maps of different scales based on the image to be detected, and each feature map is input into the corresponding task head network, with one task head network corresponding to one feature map; Each prediction branch in the task head network processes the corresponding feature map to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human key point prediction results, and behavior prediction results corresponding to each behavior category. The various prediction results obtained by the task head network based on the same anchor point in the feature map during the processing correspond to the same human figure. The final detection result of the corresponding human image in the image to be detected is obtained based on the prediction results of the same human image output by each task head network, and each prediction result corresponds to one final detection result.

2. The multi-task detection method according to claim 1, characterized in that, The multiple prediction branches also include at least one of a human body prediction branch and a head bounding box prediction branch. Correspondingly, the multiple prediction results also include at least one of a human body prediction result and a head bounding box prediction result. The behavior prediction branch is also used to predict occluded human key points, and correspondingly, the multiple prediction results also include human key point occlusion prediction results.

3. The multi-task detection method according to claim 1, characterized in that, Also includes: When the behavior prediction branch in the task head network processes the corresponding feature map, the key point features obtained by the human key point prediction branch in the same task head network based on the feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the feature map, so as to obtain the behavior prediction result corresponding to each behavior category based on the concatenated features.

4. The multi-task detection method according to claim 2, characterized in that, The human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input feature map. The first regression module is used to regress the human bounding box variables to obtain the human bounding box prediction result. The first regression module uses distributed focus loss. The human key point prediction branch consists of a second convolution module and a second regression module. The second convolution module is used to predict the human key point variables corresponding to each human image based on the input feature map. The second regression module is used to regress the human key point variables to obtain the human key point prediction result. The second regression module adopts bidirectional distributed focus loss. When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each portrait based on the input feature map. The third regression module is used to regress the head bounding box variables to obtain the head bounding box prediction result. The third regression module adopts a bidirectional distributed focus loss.

5. The multi-task detection method according to claim 4, characterized in that, The bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization digits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is the human keypoint variable, and the variable to be regressed corresponding to the third regression module is the head frame variable. The bias coefficient is determined according to the discretization digits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is the human keypoint variable, the result corresponds to the human keypoint prediction result. When the variable to be regressed is the head frame variable, the result corresponds to the head frame prediction result.

6. A training method for a multi-task detection model, characterized in that, include: Obtain a training dataset containing multiple training images. Each portrait in each training image has multiple corresponding ground truth labels, including human bounding box labels, human keypoint labels, and behavior labels under multiple behavior categories. The training images are input into a multi-task detection model, which includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and an action prediction branch. The feature extraction network obtains multiple training feature maps of different scales based on the training image, and each training feature map is input into the corresponding task head network, with one task head network corresponding to one training feature map; Each prediction branch in the task head network processes the corresponding training feature map to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. The various training results predicted based on the same anchor point in the training feature map during the task head network processing correspond to the same human image. Based on the various training results and the real labels corresponding to each training result, a variety of training losses are constructed for the corresponding task head network. The various training losses include human body bounding box loss, human body keypoint loss, and behavior loss corresponding to each of the behavior categories. The model parameters of the multi-task detection model are updated based on the various training losses corresponding to each task head network, so as to train the multi-task detection model.

7. The training method according to claim 6, characterized in that, The multiple prediction branches also include at least one of a human prediction branch and a head bounding box prediction branch. Correspondingly, the multiple ground truth labels also include at least one of a human body label and a head bounding box label. The multiple training results also include at least one of a human body training result and a head bounding box training result. The multiple training losses also include at least one of a human body loss and a head bounding box loss. The behavior prediction branch is also used to predict occluded human keypoints. Accordingly, the multiple ground truth labels also include human keypoint occlusion labels, the multiple training results also include human keypoint occlusion training results, and the multiple training losses also include human keypoint occlusion losses.

8. The training method according to claim 7, characterized in that, In the training images, multiple ground truth labels corresponding to the same portrait are assigned to the same anchor point in the training feature map, and each anchor point is assigned multiple ground truth labels for only one portrait.

9. The training method according to claim 6 or 8, characterized in that, The behavior training results corresponding to the behavior category include the prediction score of the behavior prediction branch for each behavior under the corresponding behavior category. Specifically, each anchor point corresponding to the same portrait in the training feature map has a corresponding prediction score for each behavior. Correspondingly, the process of determining the behavior loss corresponding to a behavior category includes: Based on each predicted score under the behavior category and the corresponding behavior label, the natural log loss corresponding to each predicted score is obtained; The natural log loss corresponding to each predicted score is multiplied by the category weight corresponding to the corresponding behavior label to obtain the first multiplication result corresponding to each predicted score. Each behavior under the behavior category has a corresponding category weight. The more behavior labels corresponding to the behavior are in each training image, the smaller the corresponding category weight. The first multiplication result corresponding to each predicted score under the same anchor point is added together and then multiplied with the matching score corresponding to the anchor point to obtain the second multiplication result corresponding to the anchor point. The matching score refers to the matching score between the anchor point and the behavior label under the behavior category. The sum of the second multiplication results of each anchor point corresponding to the same portrait is added together and then divided by the sum of the matching scores of each anchor point corresponding to the portrait to obtain the behavior loss of the portrait under the behavior category.

10. The training method according to claim 6, characterized in that, Also includes: When the behavior prediction branch in the task head network processes the corresponding training feature map, the key point features obtained by the human key point prediction branch in the same task head network based on the training feature map are concatenated with the behavior features obtained by the behavior prediction branch based on the training feature map, so as to obtain the behavior training result corresponding to each behavior category based on the concatenated features.

11. The training method according to claim 7, characterized in that, The human bounding box prediction branch consists of a first convolutional module and a first regression module. The first convolutional module is used to predict the human bounding box variables corresponding to each human image based on the input training feature map. The first regression module is used to regress the human bounding box variables to obtain the human bounding box training results. The first regression module adopts distributed focus loss. The human keypoint prediction branch consists of a second convolution module and a second regression module. The second convolution module is used to predict the human keypoint variables corresponding to each portrait based on the input training feature map. The second regression module is used to regress the human keypoint variables to obtain the human keypoint training results. The second regression module adopts bidirectional distributed focus loss. When the multiple prediction branches also include a head bounding box prediction branch, the head bounding box prediction branch consists of a third convolutional module and a third regression module. The third convolutional module is used to predict the head bounding box variables corresponding to each portrait based on the input training feature map. The third regression module is used to regress the head bounding box variables to obtain the head bounding box training results. The third regression module adopts a bidirectional distributed focus loss.

12. The training method according to claim 11, characterized in that, The bidirectional distributed focus loss is based on the distributed focus loss with the addition of a bias coefficient and prediction granularity. When using the bidirectional distributed focus loss to regress the variable to be regressed, the bias coefficient is used to move the position of the anchor point used in the regression process so that the moved anchor point is located at the center of the discretization digits of the variable to be regressed. The variable to be regressed corresponding to the second regression module is the human keypoint variable, and the variable to be regressed corresponding to the third regression module is the head box variable. The bias coefficient is determined according to the discretization digits of the variable to be regressed. The prediction granularity is used to adjust the accuracy of the result obtained after regressing the variable to be regressed. When the variable to be regressed is the human keypoint variable, the result corresponds to the human keypoint training result. When the variable to be regressed is the head box variable, the result corresponds to the head box training result.

13. A multi-task detection device, characterized in that, include: The first acquisition unit is used to acquire the image to be detected; The first input unit is used to input the image to be detected into a multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches. The multiple prediction branches include a human bounding box prediction branch, a human keypoint prediction branch, and a behavior prediction branch. The behavior prediction branch is used to predict behaviors under multiple behavior categories. The first feature extraction unit is used to obtain multiple feature maps of different scales from the image to be detected by the feature extraction network and input each feature map into the corresponding task head network, with one task head network corresponding to one feature map. The first prediction unit is used to process the corresponding feature maps by each prediction branch in the task head network to obtain multiple prediction results for each detected human figure in the image to be detected. The multiple prediction results include human bounding box prediction results, human key point prediction results, and behavior prediction results corresponding to each behavior category. The various prediction results obtained by the task head network based on the same anchor point in the feature map during the processing correspond to the same human figure. The result determination unit is used to obtain the final detection result of the corresponding human image in the image to be detected based on the prediction results of the same human image output by each task head network, wherein each prediction result corresponds to one final detection result.

14. A training device for a multi-task detection model, characterized in that, include: The second acquisition unit is used to acquire a training dataset, which contains multiple training images. Each portrait in each training image has multiple corresponding real labels, including human body bounding box labels, human body key point labels, and behavior labels under multiple behavior categories. The second input unit is used to input the training image into a multi-task detection model. The multi-task detection model includes a feature extraction network and multiple task head networks. Each task head network includes multiple prediction branches, including a human bounding box prediction branch, a human keypoint prediction branch, and an action prediction branch. The second feature extraction unit is used to obtain multiple training feature maps of different scales from the training image by the feature extraction network and input each training feature map into the corresponding task head network, with one task head network corresponding to one training feature map. The second prediction unit is used to process the corresponding training feature maps by each prediction branch in the task head network to obtain multiple training results for each detected human image in the training image. The multiple training results include human bounding box training results, human keypoint training results, and behavior training results corresponding to each behavior category. The various training results predicted based on the same anchor point in the training feature map during the task head network processing correspond to the same human image. The loss construction unit is used to construct multiple training losses corresponding to the task head network based on the multiple training results and the real labels corresponding to each training result. The multiple training losses include human body bounding box loss, human body keypoint loss, and behavior loss corresponding to each of the behavior categories. The parameter update unit is used to update the model parameters of the multi-task detection model based on the multiple training losses corresponding to each task head network, so as to train the multi-task detection model.

15. A multi-task detection device, characterized in that, include: One or more processors and memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task detection method as described in any one of claims 1-5.

16. A training device for a multi-task detection model, characterized in that, include: One or more processors and memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the multi-task detection model as described in any one of claims 6-12.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-task detection method as described in any one of claims 1-5.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the multi-task detection model as described in any one of claims 6-12.

Images