Human body and face combined detection method and device
By training a teacher model and generating heatmaps, a student model for human and face detection is constructed. Joint detection is performed using a loss function, which solves the problem of high manual annotation costs in joint training of human and face detection and achieves cost reduction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN XUMI YUNTU SPACE TECH CO LTD
- Filing Date
- 2022-11-03
- Publication Date
- 2026-05-22
AI Technical Summary
The high cost of manually annotating faces and bodies in joint training of human bodies and faces.
The teacher model is trained by acquiring first human body data and first face data. Heatmaps are generated using these data to construct a student model for human body and face detection. The model is then trained using a loss function for joint detection.
This reduces the cost of joint training of human bodies and faces, and decreases the need for manual annotation.
Smart Images
Figure CN115731623B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of machine learning technology, and in particular to a method and apparatus for joint detection of human body and face. Background Technology
[0002] Currently, joint face and body detection often requires manual annotation of both faces and bodies before training the detection model. However, manual annotation is extremely time-consuming and labor-intensive. Because face and body detection are common, there is a large amount of separate annotated data for both faces and bodies (including public and private datasets). However, there is currently no method for joint training of face and body detection using these separate annotated datasets.
[0003] In realizing the concept disclosed herein, the inventors discovered at least the following technical problems in the related technologies: the high cost of manually annotating the face and body in the joint training of human body and face. Summary of the Invention
[0004] In view of this, the present disclosure provides a method, apparatus, electronic device and computer-readable storage medium for joint detection of human body and face, so as to solve the problem of high cost of manual joint annotation of human face and human body in the prior art.
[0005] A first aspect of this disclosure provides a method for joint detection of human body and face, comprising: acquiring first human body data, second human body data, first face data, and second face data; training a human body detection teacher model using the first human body data, and training a face detection teacher model using the first face data; inputting the second human body data into the trained human body detection teacher model, and outputting a first heatmap of the human body center point and a second heatmap of the distances from the human body center point to each border of the human body detection box; inputting the second face data into the trained face detection teacher model, and outputting a third heatmap of the face center point and a fourth heatmap of the distances from the face center point to each border of the face detection box; determining a fifth heatmap of the human body and face center point based on the first heatmap and the third heatmap, and determining a sixth heatmap of the distances from the human body and face center point to each border of the human body and face detection box based on the second heatmap and the fourth heatmap; constructing a human body and face detection student model, and training the human body and face detection student model using a first loss function based on the fifth heatmap and the sixth heatmap; and performing joint detection of human body and face using the trained human body and face detection student model.
[0006] A second aspect of this disclosure provides a joint human body and face detection device, comprising: an acquisition module configured to acquire first human body data, second human body data, first face data, and second face data; a first training module configured to train a human body detection teacher model using the first human body data and a face detection teacher model using the first face data; and a teacher model module configured to input the second human body data into the trained human body detection teacher model, output a first heatmap of the human body center point and a second heatmap of the distances from the human body center point to each border of the human body detection box, and input the second face data into the trained face detection teacher model. The system outputs a third heatmap of the face center point and a fourth heatmap showing the distances from the face center point to each border of the face detection box; a determination module is configured to determine a fifth heatmap of the human body and face center point based on the first and third heatmaps, and a sixth heatmap showing the distances from the human body and face center point to each border of the human body and face detection box based on the second and fourth heatmaps; a second training module is configured to construct a human body and face detection student model, and train the human body and face detection student model using a first loss function based on the fifth and sixth heatmaps; and a joint detection module is configured to perform joint detection of human body and face using the trained human body and face detection student model.
[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0009] The beneficial effects of this disclosure embodiment compared with the prior art are as follows: because this disclosure embodiment acquires first human body data, second human body data, first face data and second face data; trains a human body detection teacher model using the first human body data, trains a face detection teacher model using the first face data; inputs the second human body data into the trained human body detection teacher model, outputs a first heat map of the human body center point and a second heat map of the distance from the human body center point to each border of the human body detection box; inputs the second face data into the trained face detection teacher model, outputs a third heat map of the face center point and a fourth heat map of the distance from the face center point to each border of the face detection box; A fifth heatmap is determined based on the first and third heatmaps to identify the center points of the human body and face. A sixth heatmap is determined based on the second and fourth heatmaps to identify the distances from the center points of the human body and face to the bounding boxes of the human body and face detection boxes. A human body and face detection student model is constructed and trained using the first loss function based on the fifth and sixth heatmaps. The trained human body and face detection student model is then used for joint detection of human bodies and faces. Therefore, by adopting the above-mentioned technical means, the problem of high cost of manual joint annotation of human bodies and faces in the existing technology can be solved, thereby reducing the cost of joint training of human bodies and faces. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;
[0012] Figure 2 This is a schematic flowchart of a method for joint detection of human body and face provided in an embodiment of this disclosure;
[0013] Figure 3 This is a schematic diagram of the structure of a human body and face joint detection device provided in an embodiment of this disclosure;
[0014] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.
[0016] A method and apparatus for joint detection of human body and face according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.
[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 101, 102, and 103, server 104, and network 105.
[0018] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays that support communication with server 104, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.
[0019] Server 104 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 104 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This embodiment of the disclosure does not impose any limitations on these aspects.
[0020] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be various electronic devices that provide various services to terminal devices 101, 102, and 103. When server 104 is software, it can be multiple software programs or software modules that provide various services to terminal devices 101, 102, and 103, or it can be a single software program or software module that provides various services to terminal devices 101, 102, and 103. This disclosure does not limit the scope of the embodiments.
[0021] Network 105 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), Infrared, etc. This disclosure does not limit the scope of the network.
[0022] Users can establish a communication connection with server 104 via network 105 through terminal devices 101, 102, and 103 to receive or send information, etc. It should be noted that the specific types, quantities, and combinations of terminal devices 101, 102, and 103, server 104, and network 105 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any limitations on this.
[0023] Figure 2 This is a schematic flowchart of a method for joint detection of human body and face provided in an embodiment of this disclosure. Figure 2 The method for joint detection of human body and face can be developed by Figure 1 The computer or server, or the software on the computer or server, executes the command. For example... Figure 2 As shown, the method for joint detection of human body and face includes:
[0024] S201, acquire first human body data, second human body data, first face data, and second face data;
[0025] S202, Use the first human body data to train the human body detection teacher model, and use the first face data to train the face detection teacher model;
[0026] S203, input the second human body data into the trained human body detection teacher model, output the first heat map of the human body center point and the second heat map of the distance from the human body center point to each border of the human body detection box; input the second face data into the trained face detection teacher model, output the third heat map of the face center point and the fourth heat map of the distance from the face center point to each border of the face detection box.
[0027] S204, determine the fifth heat map of the center point of the human body and face based on the first heat map and the third heat map, and determine the sixth heat map of the distance from the center point of the human body and face to each border of the human body and face detection box based on the second heat map and the fourth heat map.
[0028] S205, Construct a human body and face detection student model, based on the fifth and sixth heatmaps, and train the human body and face detection student model using the first loss function;
[0029] S206 utilizes a trained student model for human and face detection to perform joint detection of human bodies and faces.
[0030] Human body detection teacher models and face detection teacher models are commonly used neural network models. The feature extraction network in these models uses the RegNet network. The methods for training these models are standard training techniques and will not be elaborated upon here. The first human body data, second human body data, first face data, and second face data each have their own independent annotation results, but there are no results showing joint annotation of human bodies and faces.
[0031] The center point of the human body is the very center of the human body, and the center point of the face is the center of the face. A human detection bounding box represents a detected human body, and a face detection bounding box represents a detected face. Both human and face detection bounding boxes have four borders: top, bottom, left, and right. Therefore, both the second and fourth heatmaps include four heatmaps. For example, the second heatmap includes: heatmaps showing the distance from the human center point to the top border of the human detection bounding box, the distance from the human center point to the bottom border of the human detection bounding box, the distance from the human center point to the right border of the human detection bounding box, and the distance from the human center point to the left border of the human detection bounding box.
[0032] Merging the first and third heatmaps yields the fifth heatmap, and merging the second and fourth heatmaps yields the sixth heatmap. Heatmap merging can be understood as matrix stacking (heatmaps are essentially matrices).
[0033] According to the technical solution provided in this disclosure, first human body data, second human body data, first face data, and second face data are obtained; a human body detection teacher model is trained using the first human body data, and a face detection teacher model is trained using the first face data; the second human body data is input into the trained human body detection teacher model, and a first heatmap of the human body center point and a second heatmap of the distance from the human body center point to each border of the human body detection box are output; the second face data is input into the trained face detection teacher model, and a third heatmap of the face center point and a fourth heatmap of the distance from the face center point to each border of the face detection box are output; based on the first heatmap and the second heatmap... The third heatmap determines the fifth heatmap of the center points of the human body and face. The sixth heatmap, based on the second and fourth heatmaps, determines the distances from the center points of the human body and face to the bounding boxes of the human body and face detection boxes. A human body and face detection student model is constructed and trained using the first loss function based on the fifth and sixth heatmaps. The trained human body and face detection student model is then used for joint detection of human bodies and faces. Therefore, by adopting the above-mentioned technical means, the problem of high cost of manual joint annotation of human bodies and faces in the joint training of human bodies and faces in the existing technology can be solved, thereby reducing the cost of joint training of human bodies and faces.
[0034] The human body and face detection student model includes: a feature extraction network, a first-head network for predicting the center points of the human body and face, and a second-head network for predicting the distances from the center points of the human body and face to the bounding boxes of the human body and face detection boxes; wherein the first-head network and the second-head network are connected in parallel after the feature extraction network.
[0035] The feature extraction network can be a regnet network.
[0036] In step S205, a human body and face detection student model is trained using a first loss function based on the fifth and sixth heatmaps. This includes: inputting second human body data and second face data into the human body and face detection student model, and outputting a seventh heatmap of the center points of the human body and face, and an eighth heatmap of the distances from the center points of the human body and face to each border of the human body and face detection box. When the human body and face detection student model outputs the seventh and eighth heatmaps based on the second human body data and second face data, the first loss function is used to constrain the human body and face detection student model so that the difference between the fifth and seventh heatmaps and the difference between the sixth and eighth heatmaps are minimized.
[0037] The first loss function constrains the human body and face detection student model to minimize the difference between the fifth and seventh heatmaps and the difference between the sixth and eighth heatmaps. In other words, it makes the fifth heatmap close to the seventh heatmap and the sixth heatmap close to the eighth heatmap.
[0038] The first loss function includes:
[0039]
[0040] N is the number of samples in the second human body data and the second face data, and i is the sample index. i When it is the seventh heatmap, then For the fifth heatmap, L mse The difference between the fifth and seventh heatmaps is given when yi is the eighth heatmap. For the sixth heatmap, L mse This represents the difference between the sixth and eighth heatmaps.
[0041] L mse When calculating the difference between the fifth and seventh heatmaps, use L represents mse L mse When the difference is between the sixth and eighth heatmaps, L represents mse .
[0042] After constructing the human body and face detection student model, the method further includes: obtaining the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: the ninth heatmap of the center point of the human body and face and the tenth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; inputting the second human body data and the second face data into the human body and face detection student model, and outputting the seventh heatmap of the center point of the human body and face and the eighth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; when the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, the second loss function is used to constrain the human body and face detection student model so that the difference between the ninth heatmap and the seventh heatmap and the difference between the tenth heatmap and the eighth heatmap are minimized.
[0043] The second loss function includes:
[0044]
[0045] N is the number of samples in the second human body data and the second face data, and i is the sample index. i When it is the seventh heatmap, then For the ninth heatmap, L gfl The difference between the ninth and seventh heatmaps, when y i This is the eighth heatmap. The tenth heatmap, L gfl α represents the difference between the tenth and eighth heatmaps, and β and a are hyperparameters.
[0046] L gfl The difference between the ninth and seventh heatmaps is expressed as... L represents gfl L gfl The difference between the tenth and eighth heatmaps is expressed as... L represents gfl .
[0047] y i =1, indicating that y i The corresponding sample is a face sample, at which point y i This is the seventh heatmap; otherwise, it represents y. i The corresponding sample is a human body sample, at which point y i This is the eighth heatmap.
[0048] Optionally, the annotation results corresponding to the second human body data and the second face data can also be regarded as the corresponding labels of the samples, so For the label, y i The student model for human and face detection identifies the seventh or eighth heatmap.
[0049] After constructing the human body and face detection student model, the method further includes: training the human body and face detection student model using a first loss function and a second loss function, including: obtaining the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: a ninth heatmap of the center point of the human body and face and a tenth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; inputting the second human body data and the second face data into the human body and face detection student model, and outputting a seventh heatmap of the center point of the human body and face and an eighth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; when the human body and face detection student model outputs the seventh and eighth heatmaps based on the second human body data and the second face data, constraining the human body and face detection student model using the first loss function to minimize the difference between the fifth and seventh heatmaps and the difference between the sixth and eighth heatmaps, and constraining the human body and face detection student model using the second loss function to minimize the difference between the ninth and seventh heatmaps and the difference between the tenth and eighth heatmaps.
[0050] Constraining the human and face detection student model using the first and second loss functions can be understood as constraining the human and face detection student model using the total loss function, which is:
[0051]
[0052] λ1 and λ2 are hyperparameters.
[0053] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.
[0054] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.
[0055] Figure 3 This is a schematic diagram of a human body and face joint detection device provided in an embodiment of this disclosure. Figure 3 As shown, the human body and face joint detection device includes:
[0056] The acquisition module 301 is configured to acquire first human body data, second human body data, first face data, and second face data;
[0057] The first training module 302 is configured to train a human detection teacher model using the first human body data and to train a face detection teacher model using the first face data.
[0058] The teacher model module 303 is configured to input the second human body data into the trained human body detection teacher model and output a first heat map of the human body center point and a second heat map of the distance from the human body center point to each border of the human body detection box; input the second face data into the trained face detection teacher model and output a third heat map of the face center point and a fourth heat map of the distance from the face center point to each border of the face detection box.
[0059] The determination module 304 is configured to determine a fifth heat map based on the first heat map and the third heat map, and to determine a sixth heat map based on the second heat map and the fourth heat map, which shows the distances from the center point of the human body and face to each border of the human body and face detection box.
[0060] The second training module 305 is configured to build a human body and face detection student model, and train the human body and face detection student model based on the fifth heatmap and the sixth heatmap using the first loss function.
[0061] The joint detection module 306 is configured to perform joint detection of human bodies and faces using a trained human body and face detection student model.
[0062] Human body detection teacher models and face detection teacher models are commonly used neural network models. The feature extraction network in the neural network model uses the regnet network. The methods for training human body detection teacher models and face detection teacher models are common model training techniques, which will not be elaborated here.
[0063] The center point of the human body is the very center of the human body, and the center point of the face is the center of the face. A human detection bounding box represents a detected human body, and a face detection bounding box represents a detected face. Both human and face detection bounding boxes have four borders: top, bottom, left, and right. Therefore, both the second and fourth heatmaps include four heatmaps. For example, the second heatmap includes: heatmaps showing the distance from the human center point to the top border of the human detection bounding box, the distance from the human center point to the bottom border of the human detection bounding box, the distance from the human center point to the right border of the human detection bounding box, and the distance from the human center point to the left border of the human detection bounding box.
[0064] Merging the first and third heatmaps yields the fifth heatmap, and merging the second and fourth heatmaps yields the sixth heatmap. Heatmap merging can be understood as matrix stacking (heatmaps are essentially matrices).
[0065] According to the technical solution provided in this disclosure, first human body data, second human body data, first face data, and second face data are obtained; a human body detection teacher model is trained using the first human body data, and a face detection teacher model is trained using the first face data; the second human body data is input into the trained human body detection teacher model, and a first heatmap of the human body center point and a second heatmap of the distance from the human body center point to each border of the human body detection box are output; the second face data is input into the trained face detection teacher model, and a third heatmap of the face center point and a fourth heatmap of the distance from the face center point to each border of the face detection box are output; based on the first heatmap and the second heatmap... The third heatmap determines the fifth heatmap of the center points of the human body and face. The sixth heatmap, based on the second and fourth heatmaps, determines the distances from the center points of the human body and face to the bounding boxes of the human body and face detection boxes. A human body and face detection student model is constructed and trained using the first loss function based on the fifth and sixth heatmaps. The trained human body and face detection student model is then used for joint detection of human bodies and faces. Therefore, by adopting the above-mentioned technical means, the problem of high cost of manual joint annotation of human bodies and faces in the joint training of human bodies and faces in the existing technology can be solved, thereby reducing the cost of joint training of human bodies and faces.
[0066] The human body and face detection student model includes: a feature extraction network, a first-head network for predicting the center points of the human body and face, and a second-head network for predicting the distances from the center points of the human body and face to the bounding boxes of the human body and face detection boxes; wherein the first-head network and the second-head network are connected in parallel after the feature extraction network.
[0067] The feature extraction network can be a regnet network.
[0068] Optionally, the second training module 305 is further configured to input the second human body data and the second face data into the human body and face detection student model, and output a seventh heatmap of the center point of the human body and face and an eighth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; when the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, the first loss function is used to constrain the human body and face detection student model so that the difference between the fifth heatmap and the seventh heatmap and the difference between the sixth heatmap and the eighth heatmap are minimized.
[0069] The first loss function constrains the human body and face detection student model to minimize the difference between the fifth and seventh heatmaps and the difference between the sixth and eighth heatmaps. In other words, it makes the fifth heatmap close to the seventh heatmap and the sixth heatmap close to the eighth heatmap.
[0070] The first loss function includes:
[0071]
[0072] N is the number of samples in the second human body data and the second face data, and i is the sample index. i When it is the seventh heatmap, then For the fifth heatmap, L mse The difference between the fifth and seventh heatmaps is given when yi is the eighth heatmap. For the sixth heatmap, L mse This represents the difference between the sixth and eighth heatmaps.
[0073] L mse When calculating the difference between the fifth and seventh heatmaps, use L represents mse L mse When the difference is between the sixth and eighth heatmaps, L represents mse .
[0074] Optionally, the second training module 305 is further configured to obtain the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: a ninth heatmap of the center point of the human body and face and a tenth heatmap of the distances from the center point of the human body and face to each border of the human body and face detection box; inputting the second human body data and the second face data into the human body and face detection student model, and outputting a seventh heatmap of the center point of the human body and face and an eighth heatmap of the distances from the center point of the human body and face to each border of the human body and face detection box; when the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, using a second loss function to constrain the human body and face detection student model, so that the difference between the ninth heatmap and the seventh heatmap and the difference between the tenth heatmap and the eighth heatmap are minimized.
[0075] The second loss function includes:
[0076]
[0077] N is the number of samples in the second human body data and the second face data, and i is the sample index. i When it is the seventh heatmap, then For the ninth heatmap, L gfl The difference between the ninth and seventh heatmaps, when y i This is the eighth heatmap. The tenth heatmap, L gfl α represents the difference between the tenth and eighth heatmaps, and β and a are hyperparameters.
[0078] L gfl The difference between the ninth and seventh heatmaps is expressed as... L represents gfl Lgfl The difference between the tenth and eighth heatmaps is expressed as... L represents gfl .
[0079] Optionally, the second training module 305 is further configured to train a human body and face detection student model using a first loss function and a second loss function, including: obtaining the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: a ninth heatmap of the center point of the human body and face and a tenth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; inputting the second human body data and the second face data into the human body and face detection student model, and outputting a seventh heatmap of the center point of the human body and face and an eighth heatmap of the distance from the center point of the human body and face to each border of the human body and face detection box; when the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, constraining the human body and face detection student model using the first loss function to minimize the difference between the fifth heatmap and the seventh heatmap and the difference between the sixth heatmap and the eighth heatmap, and constraining the human body and face detection student model using the second loss function to minimize the difference between the ninth heatmap and the seventh heatmap and the difference between the tenth heatmap and the eighth heatmap.
[0080] Constraining the human and face detection student model using the first and second loss functions can be understood as constraining the human and face detection student model using the total loss function, which is:
[0081]
[0082] λ1 and λ2 are hyperparameters.
[0083] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.
[0084] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.
[0085] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or different components.
[0086] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0087] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 4. The memory 402 can also include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.
[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0089] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0090] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.
Claims
1. A method for joint detection of human body and face, characterized in that, include: Acquire first human body data, second human body data, first face data, and second face data; The first human body data is used to train a human body detection teacher model, and the first face data is used to train a face detection teacher model. Input the second human body data into the trained human body detection teacher model, and output the first heat map of the human body center point and the second heat map of the distance from the human body center point to each border of the human body detection box. Input the second face data into the trained face detection teacher model, and output the third heat map of the face center point and the fourth heat map of the distance from the face center point to each border of the face detection box. A fifth heat map is determined based on the first heat map and the third heat map, and a sixth heat map is determined based on the second heat map and the fourth heat map, which shows the distances from the center point of the human body and face to each border of the human body and face detection box. Construct a human body and face detection student model, and train the human body and face detection student model using the first loss function based on the fifth heatmap and the sixth heatmap; We use trained student models for human and face detection to perform joint detection of human bodies and faces.
2. The method according to claim 1, characterized in that, The human body and face detection student model includes: a feature extraction network, a first-head network for predicting the center point of the human body and face, and a second-head network for predicting the distance from the center point of the human body and face to each border of the human body and face detection box. The first head network and the second head network are connected in parallel after the feature extraction network.
3. The method according to claim 1, characterized in that, The step of training a human body and face detection student model based on the fifth and sixth heatmaps using a first loss function includes: Input the second human body data and the second face data into the human body and face detection student model, and output the seventh heat map of the center point of the human body and face and the eighth heat map of the distance from the center point of the human body and face to each border of the human body and face detection box. When the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, the first loss function is used to constrain the human body and face detection student model so that the difference between the fifth heatmap and the seventh heatmap and the difference between the sixth heatmap and the eighth heatmap are minimized.
4. The method according to claim 3, characterized in that, The first loss function includes: N is the number of samples in the second human body data and the second face data, and i is the sample index. When the value corresponds to the i-th sample in the seventh heatmap, then The value corresponding to the i-th sample in the fifth heatmap. The difference between the fifth heatmap and the seventh heatmap, when This is the value corresponding to the i-th sample in the eighth heatmap. This is the value corresponding to the i-th sample in the sixth heatmap. The difference between the sixth heatmap and the eighth heatmap.
5. The method according to claim 1, characterized in that, After constructing the student model for human and face detection, the method further includes: Obtain the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: the ninth heat map of the center point of the human body and the face and the tenth heat map of the distance from the center point of the human body and the face to each border of the human body and face detection box; Input the second human body data and the second face data into the human body and face detection student model, and output the seventh heat map of the center point of the human body and face and the eighth heat map of the distance from the center point of the human body and face to each border of the human body and face detection box. When the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, the human body and face detection student model is constrained by the second loss function to minimize the difference between the ninth heatmap and the seventh heatmap and the difference between the tenth heatmap and the eighth heatmap.
6. The method according to claim 5, characterized in that, The second loss function includes: N is the number of samples in the second human body data and the second face data, and i is the sample index. When the value corresponds to the i-th sample in the seventh heatmap, then This is the value corresponding to the i-th sample in the ninth heatmap. The difference between the ninth heatmap and the seventh heatmap, when This is the value corresponding to the i-th sample in the eighth heatmap. This is the value corresponding to the i-th sample in the tenth heatmap. α is the difference between the tenth heatmap and the eighth heatmap, and β is a hyperparameter.
7. The method according to claim 1, characterized in that, After constructing the student model for human and face detection, the method further includes: Training the human body and face detection student model using the first loss function and the second loss function includes: Obtain the annotation results corresponding to the second human body data and the second face data, wherein the annotation results include: the ninth heat map of the center point of the human body and the face and the tenth heat map of the distance from the center point of the human body and the face to each border of the human body and face detection box; Input the second human body data and the second face data into the human body and face detection student model, and output the seventh heat map of the center point of the human body and face and the eighth heat map of the distance from the center point of the human body and face to each border of the human body and face detection box. When the human body and face detection student model outputs the seventh heatmap and the eighth heatmap based on the second human body data and the second face data, the first loss function is used to constrain the human body and face detection student model to minimize the difference between the fifth heatmap and the seventh heatmap and the difference between the sixth heatmap and the eighth heatmap. The second loss function is used to constrain the human body and face detection student model to minimize the difference between the ninth heatmap and the seventh heatmap and the difference between the tenth heatmap and the eighth heatmap.
8. A human body and face joint detection device, characterized in that, include: The acquisition module is configured to acquire first human body data, second human body data, first face data, and second face data; The first training module is configured to train a human detection teacher model using the first human body data and to train a face detection teacher model using the first face data. The teacher model module is configured to input the second human body data into the trained human body detection teacher model and output a first heat map of the human body center point and a second heat map of the distance from the human body center point to each border of the human body detection box; input the second face data into the trained face detection teacher model and output a third heat map of the face center point and a fourth heat map of the distance from the face center point to each border of the face detection box. The determination module is configured to determine a fifth heatmap based on the first heatmap and the third heatmap, and to determine a sixth heatmap based on the second heatmap and the fourth heatmap, which shows the distances from the center point of the human body and face to each border of the human body and face detection box. The second training module is configured to construct a human body and face detection student model, and train the human body and face detection student model based on the fifth heatmap and the sixth heatmap using the first loss function. The joint detection module is configured to perform joint detection of human bodies and faces using a trained human body and face detection student model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.