Second order network based multi-task detection method and system
Patent Information
- Application Number
- CN202510570678.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-30
- Publication Date
- 2026-10-30
AI Technical Summary
结果，整个过程需要花费较长时间，难以满足低时延高速智能舱的要求
[0028]根据本公开的实施例，代替以往的串行检测方案，通过使用公共网络模型来提取图像数据特征，然后使用轻量型网络来完成并行多任务，这可以将DMS的多个任务(例如，人脸检测、人脸关键点检测、头部姿态估计、视线估计等)在座舱中组合成一个任务。由此，当运行时，不需要对图像数据执行多次预处理操作(例如，图像裁剪、缩放、归一化等)，能够极大地节省计算资源(例如，CPU资源)的消耗。与以往的方法相比，本公开的实施例能够在实现基本相同的检测精度下，减少50％以上的运行时间。
Smart Images
Figure CN123016401A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a multi-task detection method and system based on second-order networks. Background Technology
[0002] With the increasing demands for intelligence and safety in automobiles, Driver Monitoring Systems (DMS) are becoming a new hot topic. For example, in recent years, it has been difficult for active safety systems without DMS to obtain a five-star rating from the European New Car Assessment Programme (E-NCAP). According to the EU's General Safety Regulation (GSR), all new cars sold in the European market must be equipped with driver fatigue monitoring systems.
[0003] Such driver monitoring systems often require operations such as face detection, facial landmark detection, head pose estimation, and gaze estimation. However, in previous systems, the modules performing these operations were sequential and independent, requiring multiple preprocessing operations on the data. As a result, the entire process was time-consuming and could not meet the requirements of low-latency, high-speed intelligent cockpits.
[0004] On the other hand, current multi-task networks are first-order networks with few parallel tasks and low accuracy, making it impossible to optimize algorithm design for specific business scenarios.
[0005] The above description provides only background information related to this embodiment and may contain information that does not constitute prior art. Summary of the Invention
[0006] The main aspect of this disclosure aims to provide a method and system that extracts image features by reusing public networks and uses lightweight branch networks to perform multi-task detection, thereby significantly reducing runtime while achieving substantially the same accuracy.
[0007] The aspects of this disclosure are not limited to those described above, and those skilled in the art should be able to clearly understand from the following description other aspects not mentioned herein.
[0008] According to one aspect of this disclosure, a multi-task detection method is provided, the method comprising the following steps: receiving an input image; processing the image via a first backbone network to generate a feature map of the image and detecting faces in the image; mapping the coordinates of a face bounding box associated with the face in the image back to the generated feature map to crop out a feature map of a face region corresponding to the face bounding box; and inputting the feature map of the face region into a second backbone network for subsequent multi-task parallel detection.
[0009] In some embodiments, the feature map of the image is generated in the last layer of the first backbone network.
[0010] In some embodiments, the step of processing the image via a first backbone network to generate a feature map of the image includes performing convolution and subsampling operations on the image to generate the feature map.
[0011] In some embodiments, when multiple faces exist in the image, the step of detecting faces in the image includes: detecting the face with the highest confidence level in the image.
[0012] In some embodiments, the step of cropping the feature map of the face region corresponding to the face frame includes: using a bilinear interpolation method to crop the feature map of the face region corresponding to the face frame from the feature map of the image.
[0013] In some embodiments, the step of inputting the feature map of the face region into a second backbone network for subsequent multi-task parallel detection includes: extracting features associated with multiple subsequent detection tasks from the feature map of the face region via the second backbone network, and inputting them into multiple branch networks associated with the multiple subsequent detection tasks respectively.
[0014] In some embodiments, the method further includes the step of training the plurality of branch networks based on the sum of the loss functions of the plurality of branch networks.
[0015] In some embodiments, the first backbone network is frozen when the plurality of branch networks are trained.
[0016] In some embodiments, the plurality of detection tasks include one or more of keypoint detection, head pose estimation, and gaze estimation.
[0017] In some embodiments, the first backbone network is trained separately.
[0018] According to another aspect of this disclosure, a multi-task detection system is provided, the system comprising: a face detection module configured to: receive an input image, process the image via a first backbone network to generate a feature map of the image, and detect faces in the image; a face feature extraction module configured to: map the coordinates of a face bounding box associated with the face in the image back to the generated feature map, so as to crop out the feature map of the face region corresponding to the face bounding box; and a multi-task parallel detection module configured to: process the feature map of the face region via a second backbone network for subsequent multi-task parallel detection.
[0019] In some embodiments, the feature map of the image is generated in the last layer of the first backbone network.
[0020] In some embodiments, processing the image via a first backbone network to generate a feature map of the image includes performing convolution and subsampling operations on the image to generate the feature map.
[0021] In some embodiments, when multiple faces exist in the image, detecting faces in the image includes: detecting the face with the highest confidence level in the image.
[0022] In some embodiments, cropping the feature map of the face region corresponding to the face bounding box includes: using a bilinear interpolation method to crop the feature map of the face region corresponding to the face bounding box from the feature map of the image.
[0023] In some embodiments, processing the feature map of the face region via a second backbone network for subsequent multi-task parallel detection includes: extracting features associated with multiple subsequent detection tasks from the feature map of the face region via the second backbone network, and inputting them into multiple branch networks associated with the multiple subsequent detection tasks respectively.
[0024] In some embodiments, the plurality of branch networks are trained based on the sum of their loss functions.
[0025] In some embodiments, the first backbone network is frozen when the plurality of branch networks are trained.
[0026] In some embodiments, the plurality of detection tasks include one or more of keypoint detection, head pose estimation, and gaze estimation.
[0027] In some embodiments, the first backbone network is trained separately.
[0028] According to embodiments of this disclosure, instead of conventional serial detection schemes, image data features are extracted using a common network model, and then a lightweight network is used to perform parallel multitasking. This allows multiple tasks of the DMS (e.g., face detection, facial landmark detection, head pose estimation, gaze estimation, etc.) to be combined into a single task within the cockpit. Therefore, during runtime, multiple preprocessing operations (e.g., image cropping, scaling, normalization, etc.) on the image data are not required, significantly saving computational resources (e.g., CPU resources). Compared to conventional methods, embodiments of this disclosure can reduce runtime by more than 50% while achieving essentially the same detection accuracy.
[0029] The effects of this disclosure are not limited to those described above, and those skilled in the art should be able to clearly understand from the following description other effects not mentioned herein. Attached Figure Description
[0030] Figure 1A conceptual diagram of a previous multi-task detection method is shown schematically.
[0031] Figure 2 A conceptual diagram of a multi-task detection method according to an embodiment of the present disclosure is shown schematically.
[0032] Figure 3 A simplified example of generating feature maps from an input image is illustrated schematically.
[0033] Figure 4 An example of coordinate position mapping is illustrated schematically.
[0034] Figure 5 This is a schematic configuration diagram of a multi-task detection system according to an embodiment of the present disclosure.
[0035] Figure 6 A flowchart illustrating the training phase of a multi-task detection system according to an embodiment of the present disclosure is shown schematically.
[0036] Figure 7 A flowchart illustrating the inference phase of a multi-task detection system according to an embodiment of the present disclosure is shown schematically.
[0037] Figure 8 This is a schematic block diagram of a computing device that can be used to implement the method or apparatus according to this disclosure. Detailed Implementation
[0038] The present disclosure will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are illustrated. Those skilled in the art will recognize that the described exemplary embodiments can be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description should be considered illustrative rather than restrictive. Throughout the specification, the same reference numerals denote the same elements. Furthermore, in the following description of some embodiments, detailed descriptions of known functions and configurations incorporated therein have been omitted for clarity and brevity.
[0039] Throughout the specification and claims, unless explicitly stated otherwise, the word "comprising" and variations such as "including" or "containing" shall be understood to imply inclusion of the stated element, but not to exclude any other element. Terms including ordinal numbers (e.g., first and second) are used to describe various components, but these components are not limited by these terms. These terms are used only to distinguish one component from another.
[0040] In this specification, the terms “component,” “unit,” “module,” etc., may refer to a unit capable of performing at least one of the functions or operations described herein, which may be implemented in hardware or circuitry, software, or a combination of hardware or circuitry and software.
[0041] As mentioned above, driver monitoring systems often require various detection operations, such as face detection, facial landmark detection, head pose estimation, and gaze estimation. However, in previous systems, the modules performing these operations were sequential and independent.
[0042] Figure 1 A conceptual diagram of a previous multi-task detection method is shown schematically.
[0043] like Figure 1 As shown, in previous multi-task detection methods, each operation is performed independently. Specifically, in step 1, an image containing a face (in this context, the driver's face), captured by, for example, a camera, is input into a first backbone network. The first backbone network can be, for example, MoibleNetV1. Furthermore, the first backbone network can be a convolutional neural network (CNN). The first backbone network can employ a common first-order face detection algorithm, such as YOLO, SSD, etc., to perform operations on the input image to detect faces in the image, i.e., obtain face bounding boxes.
[0044] In step 2, the image is input into a second backbone network. This second backbone network can be, for example, MobileNetV2. Alternatively, it can be a CNN. The second backbone network performs operations on the input image to detect facial landmarks.
[0045] In step 3, the face image detected in step 1 is input into a third backbone network. The third backbone network can be, for example, MobileNetV2. Alternatively, the third backbone network can be a CNN. The third backbone network performs operations on the input face image to estimate the driver's head pose.
[0046] In step 4, the face image detected in step 1 is input into a fourth backbone network. The fourth backbone network can be, for example, MobileNetV2. Alternatively, the fourth backbone network can be a CNN. The fourth backbone network performs operations on the input face image to estimate the driver's gaze.
[0047] The driver's condition can be monitored through the above detection and estimation.
[0048] Figure 2 A conceptual diagram of a multi-task detection method according to an embodiment of the present disclosure is shown schematically.
[0049] like Figure 2As shown, according to an embodiment of the multi-task detection method of this disclosure, firstly, an image containing a face, captured by, for example, a camera, is input into a first backbone network. The first backbone network can be, for example, MoibleNetV1. Furthermore, the first backbone network can be a convolutional neural network (CNN). The first backbone network can employ a common first-order face detection algorithm, such as YOLO, SSD, etc., to perform operations on the input image to detect faces in the image, i.e., obtain face bounding boxes, such as... Figure 2 As indicated by the middle arrow A.
[0050] As those skilled in the art understand, after a CNN performs a convolution operation on an image, a feature map is generated. A feature map is the output image generated by a convolutional layer after the CNN performs a convolution operation on the input image (or the output of the previous layer); it represents the features extracted from the input image after the convolution operation. In a CNN, convolutional kernels (also called filters) slide across the input image (or the feature map of the previous layer) to perform the convolution operation. Afterward, bias terms are added and an activation function (e.g., ReLU) is applied, thereby generating the feature map.
[0051] Feature maps are typically three-dimensional, consisting of height, width, and channels. Height and width represent the spatial dimensions of the feature map, while the number of channels corresponds to the number of convolutional kernels used, each generating a corresponding feature map. Feature maps from different layers capture features at different levels of abstraction. Lower layers tend to capture fine-grained features, such as edges and textures, while higher layers capture more semantic information.
[0052] Figure 3 A simplified example of generating feature maps from an input image is illustrated schematically.
[0053] like Figure 3 As shown, a convolution operation is performed on the input image to extract information. Next, a subsampling (pooling) operation is used to reduce the dimensionality of the feature map output from the previous layer, reducing computational cost and preventing overfitting. After repeatedly performing convolution and subsampling operations on the input image, a fully connected layer integrates the information from all feature maps to obtain the final output.
[0054] Return to reference Figure 2 Next, the coordinates of the face bounding box in the image are mapped back to the feature map generated by the first backbone network (specifically, the feature map generated by the last layer of the first backbone network), such as... Figure 2 The middle arrow B indicates the region of interest (ROI) feature map corresponding to the face bounding box in the feature map.
[0055] Figure 4 An example of coordinate position mapping is illustrated schematically.
[0056] like Figure 4 As shown, the size of the original image can be called the input size, and the size of the feature map generated from the original image can be called the output size. The following relationship exists between the output size and the input size:
[0057] Output size = (Input size - Core size + 2 × Padding) / Step size + 1
[0058] Here, "kernel size," "padding," and "stride" refer to the kernel size, padding, and stride used when performing a convolution operation on the original image, respectively. In one example, the kernel size could be 3×3, the padding could be 1, and the stride could be 2.
[0059] Therefore, the coordinates (x, y) of the face bounding box in the original image can be mapped to the feature map generated from the original image based on the following formula, so as to obtain the corresponding coordinates (x, y) of the face bounding box in the feature map. ′ ,y ′ ):
[0060] x ′ = x / (input size / output size)
[0061] y ′ = y / (input size / output size)
[0062] Return to reference again Figure 2 At this point, the face bounding box can be used as a reference to determine the coordinates (x, y) of the corresponding area in the feature map. ′ ,y ′ From the feature map, the region of interest (ROI) feature map corresponding to the face is cropped out, such as... Figure 2 As indicated by the middle arrow C.
[0063] Next, the face ROI feature map is input into the second backbone network. The second backbone network extracts features related to subsequent multi-task detection from the face ROI feature map and distributes these features to one or more subsequent backbone networks (which may also be referred to as branch networks in this paper), such as the third backbone network, the fourth backbone network, the fifth backbone network, etc., so as to further perform other detection tasks, such as key point detection, head pose estimation, gaze estimation, etc.
[0064] It is worth noting that, unlike previous detection methods, in the method disclosed herein, the data input to the second backbone network, and therefore to the third, fourth, and fifth backbone networks, are face ROI feature maps cropped from the feature maps of the entire original image obtained from the first backbone network. In other words, only the first backbone network performs inference on the original image once, and the resulting inference (i.e., feature maps) can be reused for multiple detection tasks (face detection, keypoint detection, head pose estimation, gaze estimation, etc.). By sharing the output of the first backbone network across different tasks, redundant operations (e.g., operations during feature map generation, such as convolution operations) can be reduced, thereby improving efficiency.
[0065] Furthermore, since the facial ROI feature map contains significantly less data than the original image, the second, third, fourth, and fifth backbone networks can employ lightweight backbone networks, such as a simplified version of MobileNetV2 (e.g., approximately 3-5 layers of neural network). As those skilled in the art will understand, lightweight backbone networks, also known as micro-neural networks, are characterized by a smaller number of parameters and lower computational cost, enabling them to be deployed on devices with limited computing resources, such as smartphones, tablets, or other embedded devices.
[0066] Table 1 below shows a comparison of the algorithm size when implementing multi-task detection using conventional methods and methods according to embodiments of this disclosure.
[0067] Previous methods The method disclosed herein Face (boundary) detection 5.1M 5.1M Line of sight estimation 3.4M 0.35M Head pose estimation 3.4M 0.21M Key point detection 3.6M 0.23M total 15.5M 5.89M
[0068] As can be seen from Table 1, since the inference results of the first backbone network are reused when performing subsequent detection tasks (e.g., gaze estimation, head pose estimation, key point detection), the subsequent networks can use very few parameters, which greatly reduces the size of the algorithm and improves efficiency.
[0069] Figure 5 This is a schematic configuration diagram of a multi-task detection system according to an embodiment of the present disclosure.
[0070] like Figure 5 As shown, the multi-task detection system 500 may include a face detection module 501, a face feature extraction module 502, and a multi-task parallel detection module 503.
[0071] The face detection module 501 receives an input image, processes the input image via a first backbone network to generate a feature map of the image, and detects faces in the image. In some embodiments, the first backbone network may be a convolutional neural network (CNN). In these embodiments, processing the input image may include performing convolution and subsampling operations on the image to generate a feature map. In some embodiments, the feature map of the image may be a feature map generated in the last layer of the first backbone network.
[0072] Furthermore, in some embodiments, when multiple faces exist in an image, the face with the highest confidence level in the image can be detected (in the context of this document, the driver's face).
[0073] In addition, in some embodiments, the first backbone network can be trained separately.
[0074] The face feature extraction module 502 maps the coordinates of the face bounding box associated with the face in the input image data back to the generated feature map, so as to crop out the feature map of the face region corresponding to the face bounding box. In some embodiments, a bilinear interpolation method can be used to crop out the feature map of the face region corresponding to the face bounding box from the feature map of the image.
[0075] The multi-task parallel detection module 503 receives the feature map of the cropped face region and processes it via a second backbone network for subsequent multi-task parallel detection. In some embodiments, features associated with multiple subsequent detection tasks can be extracted from the feature map of the face region via the second backbone network and then input into multiple branch networks associated with the multiple subsequent detection tasks.
[0076] In some embodiments, multiple branch networks can be trained based on the sum of the loss functions of multiple branch networks.
[0077] In some embodiments, when training multiple branch networks, the first backbone network can be frozen, thereby ensuring that training multiple branch networks does not affect the first backbone network.
[0078] In some embodiments, the multiple detection tasks may include one or more of keypoint detection, head pose estimation, and gaze estimation.
[0079] Figure 6 A flowchart illustrating the training phase of a multi-task detection system according to an embodiment of the present disclosure is shown schematically.
[0080] like Figure 6 As shown, in S601, the first backbone network is trained. As mentioned above, the first backbone network will be used to detect faces in the input image. In this training, it can be trained specifically for the detection of a single face.
[0081] In S602, the trained first backbone network is used to perform face detection on the input image to obtain the coordinates of the face bounding boxes.
[0082] In S603, the coordinates of the face bounding box are mapped back to the feature map generated by the last layer of the first backbone network, and the face ROI feature map is cropped. In some embodiments, bilinear interpolation can be used to crop the face ROI feature map.
[0083] In S604, the face ROI feature map is input into a second-order network. In some embodiments, the second-order network includes a second backbone network and multiple parallel branch networks (for multi-task detection).
[0084] In S605, the first backbone network is frozen, and the second-order network is trained. In some embodiments, during training, the second-order network is trained based on the sum of the loss functions of multiple branch networks.
[0085] Figure 7 A flowchart illustrating the inference phase of a multi-task detection system according to an embodiment of the present disclosure is shown schematically.
[0086] like Figure 7 As shown, in S701, the first backbone network is used to perform face detection on the input image to obtain the coordinates of the face bounding boxes. In some embodiments, when there are multiple faces in the image, the face with the highest confidence in the image can be detected (in the context of this paper, the driver's face).
[0087] In S702, the coordinates of the face bounding box are mapped back to the feature map generated by the last layer of the first backbone network, and the face ROI feature map is cropped. In some embodiments, bilinear interpolation can be used to crop the face ROI feature map.
[0088] In S703, the face ROI feature map is input into a second-order network. In some embodiments, the second-order network includes a second backbone network and multiple parallel branch networks (for multi-task detection).
[0089] In S704, multiple branch networks compute in parallel to obtain the final detection result. In some examples, the final detection result may be, for example, head pose, facial landmarks, gaze vectors, etc.
[0090] Table 2 below shows a comparison of the accuracy when performing multi-task detection using conventional methods and methods according to embodiments of this disclosure. In the tests, 232,112 images were used as training data and 23,734 images were used as test data, with a resolution of 1920x1080.
[0091]
[0092] Table 3 below shows a comparison of inference time when performing multi-task detection using conventional methods and methods according to embodiments of this disclosure. In the tests, 232,112 images were used as training data and 23,734 images were used as test data, with a resolution of 1920x1080.
[0093]
[0094] As can be seen from Tables 2 and 3, compared with the conventional methods, the method according to the embodiments of this disclosure extracts image features by reusing public networks and uses lightweight branch networks to achieve multi-task detection, which can reduce the running time by more than 50% while achieving essentially the same accuracy, thereby greatly reducing the consumption of computing resources (e.g., CPU, GPU, NPU, DSP, etc.).
[0095] It should be understood that although four operations have been illustrated above—face detection, facial landmark detection, head pose estimation, and gaze estimation—this is merely for illustrating embodiments of the present disclosure, and the technical concept of the invention is not limited thereto. Other embodiments may include more or fewer operations, and may also include other different operations, or be replaced with other different operations.
[0096] It should be understood that although the embodiments of this disclosure have been described above in the context of face detection, this disclosure is not limited thereto, and the technical concepts of this disclosure can be applied to detect other targets in images in other scenarios.
[0097] Figure 8 This is a schematic block diagram of a computing device that can be used to implement the method or apparatus according to this disclosure.
[0098] The computing device 80 may include some or all of the following: memory 800, processor 820, storage device 840, input / output (I / O) interface 860, or communication interface 880. Structurally and / or functionally, the computing device 80 may include at least some of the following: face detection module 501, face feature extraction module 502, and multi-task parallel detection module 503. The computing device 80 may be a fixed computing device, such as a desktop computer, server, AI accelerator, etc., or a portable computing device, such as a laptop computer, smartphone, etc.
[0099] The memory 800 may store programs that allow the processor 820 to perform methods or operations according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor 820, and when the processor 820 executes the plurality of instructions, it may perform... Figure 6 or Figure 7 The method shown.
[0100] The memory 800 may be a single memory or multiple memories. In this case, the information required to perform the methods or operations according to various embodiments of this disclosure may be stored in a single memory or distributed across multiple memories. If the memory 800 comprises multiple memories, the multiple memories may be physically separated.
[0101] The memory 800 may include at least one of volatile memory or non-volatile memory. The volatile memory includes SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), and the non-volatile memory includes flash memory.
[0102] Processor 820 may include at least one core for executing at least one instruction. Processor 820 may execute instructions stored in memory 800. Processor 820 may be a single processor or multiple processors.
[0103] Even if the power supply to the computing device 80 is cut off, the storage device 840 retains the stored data. For example, the storage device 840 may include non-volatile memory and / or storage media such as magnetic tape, optical disc, or magnetic disk.
[0104] The program stored in storage device 840 can be loaded onto memory 800 before being executed by processor 820. Storage device 840 can store files created using a programming language, and can load programs created from those files, such as compilers, onto memory 800. Storage device 840 can store data to be processed by processor 820 and / or data processed by processor 820.
[0105] I / O interface 860 may include input devices, such as a keyboard and mouse, and may include output devices, such as a display device and a printer. Users can trigger processor 820 to execute programs and / or check processing results from processor 820 via the I / O interface.
[0106] The communication interface 880 can provide access to external networks. For example, the computing device 80 can communicate with other devices (e.g., face detection module 501, face feature extraction module 502, or multi-task parallel detection module 503) via the communication interface 880.
[0107] Each component of the apparatus or method according to this disclosure can be implemented as hardware or software, or as a combination of hardware and software. Furthermore, the function of each component can be implemented as software, and a microprocessor can be implemented to execute the software function corresponding to each component.
[0108] Various embodiments of the systems and techniques described herein can be implemented using digital electronic circuits, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. Various embodiments may include implementations having one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor, which may be a dedicated or general-purpose processor, coupled to receive and transfer data and instructions from a storage system, at least one input device, and at least one output device. The computer program (also referred to as a program, software, software application, or code) includes instructions for the programmable processor and is stored in a computer-readable recording medium.
[0109] Computer-readable storage media can include all types of storage devices on which computer-readable data can be stored. Computer-readable storage media can be non-volatile or non-transitory media, such as read-only memory (ROM), random access memory (RAM), compact optical disc ROM (CD-ROM), magnetic tape, floppy disk, or optical data storage devices. Furthermore, computer-readable storage media can further include temporary media, such as data transfer media. Moreover, computer-readable storage media can be distributed across computer systems connected via a network, and computer-readable program code can be stored and executed in a distributed manner.
[0110] Although the operations shown in the flowcharts / timing diagrams of this disclosure are performed sequentially, this is merely a description of the technical concept of one embodiment of this disclosure. In other words, those skilled in the art will understand that various modifications and changes can be made without departing from this disclosure; that is, the order shown in the flowcharts / timing diagrams can be changed, and one or more operations can be performed in parallel. Therefore, the flowcharts / timing diagrams are not limited to the time order.
[0111] Although embodiments of this disclosure have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of this disclosure. Therefore, embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of these embodiments is not limited to the description. Therefore, those skilled in the art will understand that the scope of this disclosure should not be limited to the embodiments explicitly described above, but only to the claims and their equivalents.
Claims
1. A multi-task detection method, the method comprising the following steps: Receive input image; The image is processed via a first backbone network to generate a feature map of the image and detect faces in the image; Based on the detected face, a feature map of the face region is cropped from the image; as well as The feature map of the face region is input into the second backbone network for subsequent multi-task parallel detection.
2. The method according to claim 1, wherein, The feature map of the image is generated in the last layer of the first backbone network.
3. The method according to claim 1, wherein, The step of processing the image via a first backbone network to generate a feature map of the image includes: Convolution and subsampling operations are performed on the image to generate the feature map.
4. The method according to claim 1, wherein, When multiple faces exist in the image, the steps for detecting faces in the image include: Detect the face with the highest confidence level in the image.
5. The method according to claim 1, wherein, The steps for cropping the feature map of the face region include: The feature map of the face region corresponding to the face bounding box is cropped from the feature map of the image using the bilinear interpolation method.
6. The method according to claim 1, wherein, The step of inputting the feature map of the face region into the second backbone network for subsequent multi-task parallel detection includes: Features associated with subsequent detection tasks are extracted from the feature map of the face region via the second backbone network, and then input into multiple branch networks associated with the subsequent detection tasks.
7. The method according to claim 6, further comprising the following step: The multiple branch networks are trained based on the sum of their loss functions.
8. The method according to claim 7, wherein, When training the multiple branch networks, the first backbone network is frozen.
9. The method according to claim 6, wherein, The multiple detection tasks include one or more of keypoint detection, head pose estimation, and gaze estimation.
10. The method according to claim 1, wherein, The first backbone network was trained separately.
11. A multi-task detection system, the system comprising: The face detection module is configured to: receive an input image, process the image via a first backbone network to generate a feature map of the image, and detect faces in the image; The face feature extraction module is configured to: based on the detected face, crop out the feature map of the face region from the image; and The multi-task parallel detection module is configured to process the feature map of the face region via a second backbone network for subsequent multi-task parallel detection.
12. The system according to claim 11, wherein, The feature map of the image is generated in the last layer of the first backbone network.
13. The system according to claim 11, wherein, Processing the image via a first backbone network to generate a feature map of the image includes: Convolution and subsampling operations are performed on the image to generate the feature map.
14. The system according to claim 11, wherein, When multiple faces exist in the image, detecting faces in the image includes: Detect the face with the highest confidence level in the image.
15. The system according to claim 11, wherein, The feature map of the cropped face region includes: The feature map of the face region corresponding to the face bounding box is cropped from the feature map of the image using the bilinear interpolation method.
16. The system according to claim 11, wherein, The feature maps of the face region are processed via a second backbone network for subsequent multi-task parallel detection, including: Features associated with subsequent detection tasks are extracted from the feature map of the face region via the second backbone network, and then input into multiple branch networks associated with the subsequent detection tasks.
17. The system according to claim 16, wherein, The multiple branch networks are trained based on the sum of their loss functions.
18. The system according to claim 17, wherein, When training the multiple branch networks, the first backbone network is frozen.
19. The system according to claim 16, wherein, The multiple detection tasks include one or more of keypoint detection, head pose estimation, and gaze estimation.
20. The system according to claim 11, wherein, The first backbone network was trained separately.
21. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-10.