Network training method, device, and storage medium
By combining the detection results of key points and non-key points to determine the loss and adjust the network model parameters, the problem of poor human key point detection performance under occlusion conditions is solved, and the accuracy and robustness of detection are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-10-08
- Publication Date
- 2026-06-09
AI Technical Summary
Existing network models perform poorly in detecting key points on the human body under occlusion conditions, making them difficult to apply effectively in actual production.
By acquiring sample images, a target network model is used for key point and auxiliary processing. The target loss is determined by combining the key point detection results with the non-key point detection results, and the network model parameters are adjusted based on this loss. Part detection and segmentation tasks are added to promote the model's learning of the correlation between key points and body parts.
The accuracy of keypoint detection was improved under occlusion conditions, enhancing the model's robustness and generalization ability in occluded environments.
Smart Images
Figure CN115908965B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a network training method, device, and storage medium. Background Technology
[0002] With the development of deep learning technology, various deep learning-based computer vision technologies are gradually being applied to real-world scenarios. For example, there is a wide range of applications for human keypoint detection, but current network models suffer from poor performance under conditions such as occlusion, hindering their application in actual production. Summary of the Invention
[0003] This application provides at least one network training method, device, and storage medium.
[0004] This application provides a network training method, comprising: acquiring several sample images, the sample images including a target object; processing the several sample images respectively using a target network model to obtain key point detection results of each sample image with respect to the target object and at least one auxiliary processing result, the auxiliary processing result being a non-key point detection result; determining the target loss based on the key point detection results and the non-key point detection results; and adjusting the parameters in the target network model using the target loss.
[0005] This application provides a network training device, comprising: a sample acquisition module for acquiring a plurality of sample images, wherein the sample images include a target object; a model processing module for processing the plurality of sample images using a target network model to obtain key point detection results of each sample image with respect to the target object and at least one auxiliary processing result, wherein the auxiliary processing result is a non-key point detection result; a loss determination module for determining a target loss based on the key point detection results and the non-key point detection results; and a parameter adjustment module for adjusting the parameters in the target network model using the target loss.
[0006] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the network training method described above.
[0007] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described network training method.
[0008] The above scheme processes sample images using a target network model to obtain keypoint detection results and auxiliary processing results. Then, it uses the keypoint detection results and auxiliary processing results to determine the target loss, and then adjusts the parameters in the target network model based on the target loss. Compared with directly using the loss determined by the keypoint detection results to adjust the parameters in the target network model, the target network model trained by the former can refer to non-keypoint detection results, so that the keypoint detection results obtained by the former are more accurate when the target object is occluded.
[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0011] Figure 1 This is a flowchart illustrating an embodiment of the network training method of this application;
[0012] Figure 2 This is another flowchart illustrating an embodiment of the network training method of this application;
[0013] Figure 3 This is a schematic diagram of the structure of an embodiment of the network training device of this application;
[0014] Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;
[0015] Figure 5 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0016] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0017] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0018] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the network training method of this application. Figure 1 As shown, the network training method provided in this embodiment may include the following steps:
[0020] Step S11: Obtain several sample images, which include the target object.
[0021] The "several" mentioned in the embodiments of this disclosure refers to one or more. The target object can be any object that needs to be detected for key points. For example, the target object can be a human body, an animal body, a vehicle, etc.
[0022] The sample images can be acquired by the execution device of the network training method provided in this embodiment directly photographing the target object, or by an image acquisition device that establishes a communication connection with the execution device photographing the target object, or by drawing software in the execution device or other devices. For example, the sample images can be obtained from a publicly available sample image library. No specific limitations are made here regarding the method of acquiring the sample images.
[0023] Step S12: Use the target network model to process several sample images respectively to obtain the key point detection results of the target object in each sample image and at least one auxiliary processing result. The auxiliary processing result is the non-key point detection result.
[0024] The target network model can be any model capable of keypoint detection, such as RCNN, YOLO, or SSD. The specific network structure of the target network model is not specified here. In some disclosed embodiments, the non-keypoint detection result can be the result of processing the target object in the sample image using methods other than keypoint detection.
[0025] Step S13: Determine the target loss based on the key point detection results and non-key point detection results.
[0026] A loss is determined using the keypoint detection results, and another loss is determined using the non-keypoint detection results. The two losses are then weighted and fused to obtain the target loss.
[0027] Step S14: Use the target loss to adjust the parameters in the target network model.
[0028] The method of adjusting the parameters in the network model using loss can refer to the conventional adjustment method, and will not be described in detail here.
[0029] The above scheme processes sample images using a target network model to obtain keypoint detection results and auxiliary processing results. Then, it uses the keypoint detection results and auxiliary processing results to determine the target loss, and then adjusts the parameters in the target network model based on the target loss. Compared with directly using the loss determined by the keypoint detection results to adjust the parameters in the target network model, the target network model trained by the former can refer to non-keypoint detection results, so that the keypoint detection results obtained by the former are more accurate when the target object is occluded.
[0030] In some disclosed embodiments, the auxiliary processing result includes the region detection result of the target region. The target region is the region of the target object in the sample image. For example, the target region may be a region containing all parts of the target object, or a region containing only some parts of the target object. Optionally, the target region includes a first region and / or a second region. The first region is the region in the sample image to which the bounding box of each part of the target object belongs. The second region is the segmented region corresponding to each part. The keypoint detection result includes the positions of several keypoints in the sample image. Each keypoint corresponds to a part. For example, the first region may be the position of the bounding box corresponding to each part obtained by performing part detection on the sample image, and the second region may be the segmented region corresponding to each part obtained by segmenting the sample image. For example, the bounding box may be a rectangle, containing one rectangle corresponding to each part. For example, the target object is a human body, and the parts of the human body may be the head, torso, left arm (including hand), right arm (including hand), left leg (including foot), right leg (including foot), etc. Key points of the human body can be the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Of course, these key points are just examples. If more detailed key points are needed, they can be further refined and added.
[0031] Step S13 above may include the following steps:
[0032] Using the keypoint detection results, the keypoint loss is determined. Then, using the keypoint and region detection results, the first relative positional relationship between each keypoint and the target region is determined. Based on this first relative positional relationship, the first loss between the keypoint detection results and the region detection results is determined. Finally, the keypoint loss and the first loss are weighted and fused to determine the target loss.
[0033] The method for determining keypoint loss using keypoint detection results can be a conventional method, which will not be elaborated upon here. Optionally, the first relative positional relationship can include a first relative distance. The method for determining the first relative positional relationship between each keypoint and the target region using keypoint detection results and region detection results can be:
[0034] For each keypoint, in response to the keypoint being within the corresponding target area, a first relative distance between the keypoint and the target area is determined as a first preset value. For example, the first preset value can be 0. For each keypoint, in response to the keypoint being outside the corresponding target area, the shortest distance between the keypoint and the target area is taken as the first relative distance between the keypoint and the target area.
[0035] For example, the target area includes a first area. For each key point, in response to the key point being within the corresponding first area, a first relative distance between the key point and the first area is determined as a first preset value. For each key point, in response to the key point being outside the corresponding first area, the shortest distance between the key point and the first area is taken as the first relative distance between the key point and the first area.
[0036] For example, the target area includes a second area. For each key point, in response to the key point being within the corresponding second area, a first relative distance between the key point and the second area is determined as a first preset value. For each key point, in response to the key point being outside the corresponding second area, the shortest distance between the key point and the second area is taken as the first relative distance between the key point and the second area.
[0037] For example, the target area includes a first area and a second area. The first relative distance between each key point and the first area, and the first relative distance between each key point and the second area are determined respectively. The specific methods for determining the first relative distance between each key point and the first area, and the methods for determining the first relative distance between each key point and the second area, can be referred to the above, and will not be repeated here.
[0038] In some disclosed embodiments, the first loss includes a first sub-loss related to a first region and a second sub-loss related to a second region. Specifically, the method for determining the first loss between the keypoint detection results and the region detection results based on the first relative positional relationship corresponding to each keypoint can be:
[0039] The sum of the first relative distances between each keypoint and its corresponding first region is used as the first sub-loss. The sum of the first relative distances between each keypoint and its corresponding second region is used as the second sub-loss. For example, the sum of the first relative distances related to the first region is used as the first sub-loss, and the sum of the second relative distances related to the second region is used as the second sub-loss. In some application scenarios, the minimum first relative distance related to the first region and the minimum first relative distance related to the second region can be used directly as the first sub-loss and the second sub-loss. For example, if there are three keypoints, each belonging to one of the three first regions and one of the three second regions, and each keypoint is outside of one of the first regions and one of the two second regions, and there are three first relative distances related to the first regions, the minimum first relative distance is used as the first sub-loss; similarly, if there are three first relative distances related to the second regions, the minimum first relative distance is used as the second sub-loss.
[0040] For the specific method of determining the first sub-loss, please refer to formula (1), and for the method of determining the second sub-loss, please refer to formula (2).
[0041]
[0042] As shown in formula (1), when a key point is within the corresponding component detection box, the distance between the key point and the component detection box is 0; when a key point is outside the corresponding component detection box, the shortest distance between the key point and the corresponding component detection box is taken as the shortest distance. Where L match_od_kpts Let represent the first relative distance between the i-th key point and its corresponding component detection box. The sum of the first relative distances between each key point and its corresponding component detection box is taken as the first sub-loss.
[0043]
[0044] As shown in formula (2), when a key point is within the corresponding component segmentation area, the distance between the key point and the component segmentation area is 0. When a key point is outside the corresponding component segmentation area, the shortest distance between the key point and the corresponding component segmentation area is taken as the shortest distance. Wherein, L match_seg_kpts The first relative distance between the i-th key point and its corresponding component segmentation region is represented by the sum of the first relative distances between each key point and its corresponding component segmentation region, which is used as the second sub-loss.
[0045] In some disclosed embodiments, the network training method may further include the following steps:
[0046] Based on the second relative positional relationship between the first and second regions corresponding to each part, a second loss is determined. On this basis, step S13 may include the following steps: weighted fusion of the keypoint loss, the first loss, and the second loss to determine the target loss. The second relative positional relationship includes a second relative distance. Each first region corresponds to one second region. For example, the target object is a human body, and parts include the head, neck, etc. The head includes a first region and a second region; that is, there is a corresponding relationship between the first region related to the head and the second region related to the head. The same applies to other parts of the target object. The method for determining the second loss based on the second relative positional relationship between the first and second regions corresponding to each part can be:
[0047] For each pixel in each second region, in response to the pixel being within the target first region, a second relative distance between the pixel and the target first region is determined as a second preset value. Here, the target first region is the first region corresponding to the second region. For example, for each pixel in the head-related second region, in response to the pixel being within the head-related first region, the second relative distance between the pixel and the first region is the second preset value. Here, the second preset value can be 0. In response to the pixel being outside the target first region, the shortest distance between the pixel and the target first region is used as the second relative distance.
[0048] The second loss is determined by combining the second relative distances. Specifically, the second loss L is determined by combining the second relative distances. match_od_seg The method can be found in formula (3).
[0049]
[0050] Specifically, the second loss is 0 when all pixels in the second region are within the corresponding component detection box; otherwise, the sum of the distances between each pixel outside the component detection box and the detection box is used as the second loss.
[0051] In some disclosed embodiments, when the processing result includes the region detection result of the target region, a region detection loss can be determined based on the region detection result. Specifically, if the target region includes a first region, the region detection loss includes the first region detection loss. If the target region includes a second region, the region detection loss includes the second region detection loss. If the target region includes both the first and second regions, the region detection loss includes both the first and second region detection losses. The region detection loss and the keypoint detection loss are weighted and fused to determine the target loss.
[0052] Because the human keypoint prediction task aims to predict the position coordinates of major joints in the human body, simply using keypoint loss for keypoint prediction makes it difficult to learn the correlation between keypoints and body parts. This results in poor keypoint prediction performance when the target area is occluded or has similar features. Adding part detection and / or part segmentation tasks can improve the performance of each task through multi-task parameter sharing and implicit knowledge sharing.
[0053] In addition, by increasing the consistency loss between the part detection box position and the key point position, the consistency loss between the part segmentation region and the key point position, and the consistency loss between the part detection box and the part segmentation region, the model can learn the correlation between key points and body parts, thereby enhancing its robustness in special cases.
[0054] In some disclosed embodiments, the auxiliary processing results also include pose estimation results for the target object. These pose estimation results include orientation estimation results and / or posture estimation results between the target object and the imaging device. The imaging device is a device that captures sample images. For example, when the target object is a human body, the angle is 0 degrees when viewed from the front and 180 degrees when viewed from the back. The posture estimation results can represent postures such as standing, bending over, sitting, or lying down.
[0055] Step S13 above may include the following steps:
[0056] Using the pose estimation results, the pose estimation loss is determined. Then, the keypoint loss and pose estimation loss are weighted and fused to determine the target loss. In some disclosed embodiments, the pose estimation loss includes orientation estimation loss, and the keypoint loss and orientation estimation loss are weighted and fused to determine the target loss. In some disclosed embodiments, the pose estimation loss includes attitude estimation loss, and the keypoint loss and attitude estimation loss are weighted and fused to determine the target loss. In some disclosed embodiments, the pose estimation loss includes orientation estimation loss and attitude estimation loss. The method of determining the pose estimation loss using the pose estimation results can be either using the orientation estimation results to determine the orientation estimation loss, or using the attitude estimation results to determine the attitude estimation loss. Then, the method of weightedly fusing the keypoint loss and pose estimation loss to determine the target loss can be either using the keypoint loss and pose estimation loss to obtain the target loss. The orientation estimation loss can be determined based on the orientation regression ground truth and the orientation estimation detection results, and the attitude estimation loss can be determined based on the attitude classification ground truth and the attitude estimation results. The specific methods for determining the orientation estimation loss and attitude estimation loss are not the focus of this application and will not be described in detail here.
[0057] In some publicly available embodiments, the keypoint loss, first loss, second loss, region detection loss, orientation estimation loss, and pose estimation loss are weighted and fused to obtain the target loss. The first loss includes a first sub-loss and a second sub-loss, and the region detection loss includes a first region detection loss and a second region detection loss. The method for determining the target loss can be found in formula (4).
[0058] L sum =∑α i *L i Formula (4);
[0059] In formula (4), i∈[1,8] refers to the sequence number of the eight losses mentioned above (keypoint loss, first sub-loss, second sub-loss, second loss, first region detection loss, second region detection loss, orientation estimation loss, and pose estimation loss). α i Let L be the weight of the i-th loss, which is a hyperparameter and is obtained empirically or through a hyperparameter search algorithm. i Let be the loss value of the i-th loss.
[0060] In some publicly available embodiments, the target loss can be determined by combining one or more of the following: a first sub-loss, a second sub-loss, a second loss, a first region detection loss, a second region detection loss, an orientation estimation loss, and a pose estimation loss, with the keypoint loss.
[0061] Because the features of the same key point on the human body vary greatly under different poses, it is difficult to learn the feature differences under different viewpoints and human poses using key point tasks alone, resulting in poor key point detection performance in some scenarios. Adding additional human orientation and human pose classification tasks helps to learn the differences between key points and human poses through multi-task implicit knowledge sharing, thereby improving key point detection performance and generalization ability under different viewpoints and poses.
[0062] In some disclosed embodiments, the following steps may be performed before performing step S12 described above:
[0063] The sample images are annotated, specifically including target object key point annotation, component detection box annotation, component segmentation annotation, orientation regression, and pose classification. Taking the human body as an example, body component detection divides the human body into parts such as head, torso, left arm (including hand), right arm, left leg (including foot), and right leg (which can be further subdivided as needed). Body segmentation annotation divides the human body into pixel-by-pixel segments according to the above parts. Human body key points are annotated with 17 key points, specifically the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles (more refined key points can be added as needed). Human body orientation is mapped to 0-360 degree angle values, with the front view of the human body as 0 degrees and the back view as 180 degrees, increasing clockwise. Human body pose classification categorizes human body poses into upright, bent-over, sitting, and lying postures.
[0064] To better understand the network training method provided in the embodiments of this disclosure, please refer to... Figure 2 . Figure 2 This is another flowchart illustrating an embodiment of the network training method of this application. For example... Figure 2 The process involves using a skeletal network to extract features from the input sample image, and then inputting these features into the component detection branch, keypoint detection branch, component segmentation branch, orientation regression branch, and pose classification branch, respectively.
[0065] The component detection branch outputs the region detection results for each first region, the keypoint detection branch outputs the keypoint detection results, the component segmentation branch outputs the region detection results for each second region, the orientation regression branch outputs the orientation estimation results for the target object, and the pose classification branch outputs the pose estimation results for the target object. The component detection ground truth is used to determine the first region detection loss. The keypoint detection ground truth is used to determine the keypoint loss. The component segmentation ground truth is used to determine the second region detection loss. The orientation regression ground truth and pose classification ground truth are... Figure 2 (Not shown in the diagram). Combining the keypoint detection results with the region detection results of the first region determines the first sub-loss. Combining the keypoint detection results with the region detection results of the second region determines the second sub-loss. Combining the region detection results of the first region with the region detection results of the second region determines the second loss. Weighted fusion of these eight losses yields the total loss. The total loss is used to adjust the parameters in the target network model, making the detection results of each branch of the trained target network model more accurate in subsequent applications. For example, after determining the total loss, backpropagation is performed to update the parameters of the target network model until a predetermined condition is met, at which point training ends.
[0066] The trained target network model can be deployed to an execution device to perform keypoint detection on input image data. In actual deployment, the output of the target task's branch is used for prediction, and useless branches can be deleted. For example, if the target task is keypoint detection, then branches other than the keypoint detection branch can be deleted.
[0067] The above-mentioned approach, which optimizes the performance of keypoint models through multi-task optimization, improves the performance and robustness of keypoint models by leveraging implicit knowledge sharing among related tasks and by utilizing positional consistency loss from detection and segmentation tasks to promote the model's learning of the correlation between keypoints and body parts.
[0068] In addition, single-model multi-tasking allows for the deployment of multiple task branches when multiple tasks are needed simultaneously, reducing overall time consumption.
[0069] The network training method can be executed by a network training device, such as a terminal device, server, or other processing device. The terminal device can be a monitoring device in a security system, a network video recorder, user equipment (UE), mobile device, user terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the network training method can be implemented by a processor calling computer-readable instructions stored in memory.
[0070] Please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the network training device of this application. The network training device 30 includes a sample acquisition module 31, a model processing module 32, a loss determination module 33, and a parameter adjustment module 34. The sample acquisition module 31 is used to acquire a plurality of sample images, the sample images including a target object; the model processing module 32 is used to process the plurality of sample images using the target network model respectively to obtain key point detection results of each sample image with respect to the target object and at least one auxiliary processing result, the auxiliary processing result being a non-key point detection result; the loss determination module 33 is used to determine the target loss based on the key point detection results and the non-key point detection results; the parameter adjustment module 34 is used to adjust the parameters in the target network model using the target loss.
[0071] The above scheme processes sample images using a target network model to obtain keypoint detection results and auxiliary processing results. Then, it uses the keypoint detection results and auxiliary processing results to determine the target loss, and then adjusts the parameters in the target network model based on the target loss. Compared with directly using the loss determined by the keypoint detection results to adjust the parameters in the target network model, the target network model trained by the former can refer to non-keypoint detection results, so that the keypoint detection results obtained by the former are more accurate when the target object is occluded.
[0072] The functions of each module can be found in the network training method implementation examples, and will not be repeated here.
[0073] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 40 includes a memory 41 and a processor 42. The processor 42 is used to execute program instructions stored in the memory 41 to implement the steps in any of the above-described network training method embodiments. In a specific implementation scenario, the electronic device 40 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 40 may also include mobile devices such as laptops and tablets, which are not limited here.
[0074] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the above-described network training method embodiments. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.
[0075] The above scheme processes sample images using a target network model to obtain keypoint detection results and auxiliary processing results. Then, it uses the keypoint detection results and auxiliary processing results to determine the target loss, and then adjusts the parameters in the target network model based on the target loss. Compared with directly using the loss determined by the keypoint detection results to adjust the parameters in the target network model, the target network model trained by the former can refer to non-keypoint detection results, so that the keypoint detection results obtained by the former are more accurate when the target object is occluded.
[0076] Please see Figure 5 , Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described network training method embodiments.
[0077] The above scheme processes sample images using a target network model to obtain keypoint detection results and auxiliary processing results. Then, it uses the keypoint detection results and auxiliary processing results to determine the target loss, and then adjusts the parameters in the target network model based on the target loss. Compared with directly using the loss determined by the keypoint detection results to adjust the parameters in the target network model, the target network model trained by the former can refer to non-keypoint detection results, so that the keypoint detection results obtained by the former are more accurate when the target object is occluded.
[0078] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0079] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A network training method, characterized in that, include: Acquire several sample images, wherein the sample images include the target object; The target network model is used to process several sample images to obtain key point detection results of the target object for each sample image and at least one auxiliary processing result, wherein the auxiliary processing result is a non-key point detection result; Based on the key point detection results and the non-key point detection results, the target loss is determined; The parameters in the target network model are adjusted using the target loss. The auxiliary processing result includes the region detection result of the target region, where the target region is the region of the target object in the sample image, and the target region includes the regions of each part of the target object. The step of determining the target loss based on the key point detection result and the non-key point detection result includes: Using the keypoint detection results, the keypoint loss is determined; and, Using the key point detection results and the region detection results, a first relative positional relationship between each key point and the target region is determined, and based on the first relative positional relationship corresponding to each key point, a first loss between the key point detection results and the region detection results is determined; The key point loss is weighted and fused with the first loss to determine the target loss; The target region includes a first region and / or a second region. The first region is the region in the sample image to which the location detection box of each part of the target object belongs. The second region is the segmentation region corresponding to each part. The key point detection result includes the position of several key points in the sample image. Each key point corresponds to a part. The second region is obtained by pixel-by-pixel segmentation according to each part of the target object. The second loss is determined based on the second relative positional relationship between the first region and the second region corresponding to each of the aforementioned parts; The step of weightedly fusing the keypoint loss with the first loss to determine the target loss includes: The target loss is determined by weighted fusion of the key point loss, the first loss, and the second loss.
2. The method according to claim 1, characterized in that, The first relative positional relationship includes a first relative distance: wherein, Determining the first relative positional relationship between each key point and the target region using the key point detection results and the region detection results includes: For each key point, in response to the key point being within the corresponding target area, a first relative distance between the key point and the target area is determined as a first preset value; For each key point, in response to the key point being outside the corresponding target area, the shortest distance between the key point and the target area is taken as the first relative distance between the key point and the target area. And / or, the first loss includes a first sub-loss related to the first region and a second sub-loss related to the second region, and determining the first loss between the keypoint detection result and the region detection result based on the first relative positional relationship corresponding to each keypoint includes: The sum of the first relative distances between each of the key points and the corresponding first region is taken as the first sub-loss; and, The sum of the first relative distances between each key point and the corresponding second region is used as the second sub-loss.
3. The method according to claim 1, characterized in that, The second relative positional relationship includes a second relative distance, and each first region corresponds to one second region. The step of determining the second loss based on the second relative positional relationship between the first region and the second region corresponding to each of the aforementioned locations includes: For each pixel in the second region, in response to the pixel being in the target first region, a second relative distance between the pixel and the target first region is determined to be a second preset value, wherein the target first region is the first region corresponding to the second region; In response to the pixel being outside the target first region, the shortest distance between the pixel and the target first region is taken as the second relative distance; The second loss is determined by combining the second relative distances.
4. The method according to any one of claims 1-3, characterized in that, The auxiliary processing results include pose estimation results for the target object, and the determination of target loss based on the key point detection results and the non-key point detection results includes: Using the pose estimation results, determine the pose estimation loss; The target loss is determined by weighted fusion of the keypoint loss and the pose estimation loss.
5. The method according to claim 4, characterized in that, The pose estimation result includes the orientation estimation result and pose estimation result between the target object and the imaging device. The step of determining the pose estimation loss using the pose estimation result includes: Using the orientation estimation results, determine the orientation estimation loss; and, Using the attitude estimation results, determine the attitude estimation loss; The step of weightedly fusing the keypoint loss and the pose estimation loss to obtain the target loss includes: The target loss is obtained by weighted fusion of the key point loss, the orientation estimation loss, and the pose estimation loss.
6. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 5.
7. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.