Face detection methods, equipment and storage media

Abstract

This application provides a face detection method, device, and storage medium. The method divides the face detection boxes predicted by a face detection model into groups, clusters the resulting high-confidence face detection box set to obtain at least one composite line, and matches the average width and height of the composite line and its corresponding face detection boxes with the spatial positions of each face detection box in the resulting low-confidence face detection box set. If a match is successful, the corresponding face detection box is recalled to the high-confidence face detection box set, achieving face recognition of the target image and recalling missed face detection boxes, thus improving the recall rate of the face detection model, enhancing subsequent face beautification effects, and improving the user's photo-taking experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a face detection method, device and storage medium. Background Technology

[0002] With the development of terminal device technology, camera image algorithms have been widely used in terminal devices. Among them, face detection is one of the most fundamental and important core capabilities of camera image algorithms. It can supply the detected face information to upstream and downstream algorithms such as automatic exposure (AE), autofocus (AF), face super-resolution reconstruction technology (FASER), and beauty enhancement to optimize the (face) imaging effect.

[0003] In face detection models, recall represents the proportion of correctly detected faces out of all actually existing faces, and its accuracy is crucial for image processing algorithms. However, in group photo scenarios, the recall of face detection models can significantly decrease, leading to a decline in the quality of the photos. Therefore, improving the recall of face detection models is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a face detection method, device, and storage medium, aiming to improve the recall rate of the face detection model without increasing the performance requirements of electronic devices or increasing the power consumption of electronic devices, thereby improving the user's photo-taking experience.

[0005] In a first aspect, embodiments of this application provide a face detection method, the method comprising: predicting face detection boxes on an acquired current frame image based on a pre-trained face detection model to obtain a set of face detection boxes, wherein the information of each face detection box in the set includes confidence level, width, height, and first spatial coordinates; dividing the set of face detection boxes based on a first preset confidence threshold, a second preset confidence threshold, and the confidence level of each face detection box to obtain a set of high-confidence face detection boxes and a set of low-confidence face detection boxes, wherein the first preset confidence threshold is greater than the second preset confidence threshold; and processing the high-confidence face detection boxes... Each face detection box in the face detection box set is clustered to obtain at least one group photo line and its corresponding second spatial coordinates. The first average width and first average height of the face detection boxes on each group photo line are calculated. The first spatial coordinates, width, and height of each face detection box in the low-confidence face detection box set are compared with the second spatial coordinates, first average width, and first average height of its corresponding group photo line. If the comparison result satisfies the spatial positional relationship constraint of the first group photo line, the corresponding face detection box is recalled to the high-confidence face detection box set.

[0006] It should be noted that in group photos, the typical arrangement of the main subjects (people) is in rows. Therefore, in this embodiment, the row vector of the main subjects in the group photo is defined as the group photo line. For example... Figure 4 and Figure 5 In the group photo scene, the main subjects are arranged in two rows.

[0007] For example, the face detection model described above can be selected based on the group photo scenario, such as the YOLO series algorithms, R-CNN series algorithms, and / or any open-source model based on the open-source object detection toolkit (MMdetection platform) provided by the Multimedia Laboratory of the Chinese University of Hong Kong. The following embodiments use open-source models on the MMdetection platform as examples.

[0008] For example, the first confidence threshold and the second confidence threshold are used to divide the face detection box set into a high-confidence face detection box set and a low-confidence face detection box set. The first confidence threshold is greater than the second confidence threshold, and the combination of the first confidence threshold and the second confidence threshold forms a low-confidence threshold range. The first confidence threshold can be set to 0.5, and the second confidence threshold can be set to 0.3. When the confidence of a face detection box is greater than or equal to 0.5, the face detection box is classified into the high-confidence face detection box set; when the confidence of a face detection box is greater than or equal to 0.3 and less than 0.5, the face detection box is classified into the low-confidence face detection box set.

[0009] For example, the clustering algorithms mentioned above include, but are not limited to, the k-means algorithm, support vector machines (SVM), or deep learning models. The following embodiments use the k-means algorithm as an example for description.

[0010] For example, an example of the group photo line in the above group photo scenario can be found here. Figure 10 Lines HYX1 and HYX2 in the diagram.

[0011] Therefore, by dividing the face detection boxes predicted by the face detection model into groups, and clustering the resulting high-confidence face detection box set, at least one composite line is obtained. The average width and height of the composite line and its corresponding face detection box are then matched with the spatial position of each face detection box in the resulting low-confidence face detection box set. If a match is successful, the corresponding face detection box is recalled to the high-confidence face detection box set. This achieves face recognition of the target image and recall of missed face detection boxes, improving subsequent face beautification effects and enhancing the user's photo-taking experience.

[0012] According to the first aspect, the step of clustering each face detection box in the high-confidence face detection box set to obtain at least one composite line and its corresponding second spatial coordinates, and calculating the first average width value and the first average height value of the face detection box on each composite line, includes: using a clustering algorithm to cluster the y-coordinates of the first spatial coordinates of each face detection box in the high-confidence face detection box set, setting at least one cluster center line obtained by clustering as the composite line; and calculating the first average width value and the first average height value of the face detection box corresponding to each composite line.

[0013] It should be noted that when clustering face detection boxes in the high-confidence face detection box set, at least one cluster group can be obtained. Each cluster group corresponds to a cluster center line, which is the composite line of that cluster group. Therefore, each composite line corresponds to a cluster group, and the cluster group contains multiple face detection boxes. By calculating the average width and average height of these multiple face detection boxes, the average width and average height of the composite line can be obtained.

[0014] According to the first aspect, or any implementation of the first aspect above, the step of using a clustering algorithm to cluster the y-coordinates of the first spatial coordinates of each face detection box in the high-confidence face detection box set, and setting at least one cluster center line obtained by clustering as the grouping line, includes: using a k-means clustering algorithm to cluster the y-coordinates of each face detection box in the high-confidence face detection box set, and using the elbow method to segment the clustering results to obtain at least one grouping line.

[0015] According to the first aspect, or any implementation of the first aspect above, the step of recalling the corresponding face detection box to the high-confidence face detection box set when the comparison result satisfies the spatial positional relationship constraint of the first group photo line includes: obtaining the current first face detection box in the low-confidence face detection box set; determining that the current first face detection box is on the group photo line when the second spatial coordinate of any of the group photo lines falls within the first spatial coordinate of the current first face detection box, and the difference between the width of the current first face detection box and the first average width value of the group photo line is within the preset difference range corresponding to the group photo line, and the difference between the height of the current first face detection box and the first average height value of the group photo line is within the preset difference range, determining that the comparison result satisfies the spatial positional relationship constraint of the first group photo line; recalling the current first face detection box to the high-confidence face detection box set until all the first face detection boxes in the low-confidence face detection box set have been traversed.

[0016] Understandably, since the group photo line is obtained by clustering each face detection box in the set of high-confidence face detection boxes, there is a spatial positional constraint between each face detection box and its corresponding group photo line. This spatial positional constraint can be used to determine whether the face detection box is a missed real face.

[0017] For example, the aforementioned preset difference range can be less than 5%.

[0018] For example, such as Figure 10 If the face detection boxes of personnel RW2 and RW3 satisfy the above spatial positional relationship constraints with the group photo line HYX1, then the faces of personnel RW2 and RW3 can be determined as real faces that were missed.

[0019] According to the first aspect, or any implementation of the first aspect above, the preset difference range includes: obtaining the largest and smallest face detection boxes in the face detection box set; removing the largest and smallest face detection boxes from the multiple face detection boxes on each group photo line; calculating the second average width and second average height values ​​of the remaining face detection boxes on each group photo line; calculating the aspect ratio values ​​of the second average width and second average height values ​​of each group photo line with respect to the largest and smallest face detection boxes; and setting the largest aspect ratio value among the multiple aspect ratio values ​​calculated for each group photo line as the preset difference range for its corresponding group photo line.

[0020] According to the first aspect, or any implementation of the first aspect above, before clustering each face detection box in the high-confidence face detection box set to obtain at least one composite line and its corresponding second spatial coordinates, the method further includes: predicting the subject detection box of the current frame image based on a pre-trained subject detection model to obtain a subject detection box set; matching each face detection box in the low-confidence face detection box set with each subject detection box in the subject detection box set; and recalling the corresponding face detection box to the high-confidence face detection box set if the matching is successful.

[0021] It should be noted that some mobile phones have built-in face detection models that support both face detection bounding box prediction and subject detection bounding box prediction tasks; that is, the face detection model includes a sub-module for face detection and a sub-module for subject detection. Other mobile phones, however, do not support subject detection bounding box prediction; in such cases, other hot subject detection models can be used separately to predict the subject bounding box for the current image.

[0022] It is understandable that when a face detection bounding box is a true missed face, there is a spatial relationship constraint between the face detection bounding box and its corresponding subject detection bounding box. Therefore, the subject detection bounding box can be used as prior information for recalling low-confidence face detection bounding boxes. When the low-confidence face detection bounding box and any subject detection bounding box in the set of subject detection bounding boxes have this spatial relationship constraint, the face detection bounding box is determined to be a missed face detection bounding box.

[0023] For example, see Figure 6 The spatial relationship between the face detection bounding box RNK1 and the subject detection bounding box ZTK1 in the image is such that the face detection bounding box and the subject detection bounding box satisfy the following conditions: Figure 6 When the spatial relationship is shown in the diagram, the face detection box RNK1 can be identified as a real face.

[0024] For example, such as Figure 7The spatial relationship between the face detection bounding box and the subject detection bounding box of person RW1 shown can be used to identify person RW1 as a real face that was missed when the spatial relationship is fully represented.

[0025] Therefore, in this embodiment of the application, before clustering high-confidence face detection boxes to obtain the group photo line, the low-confidence face detection boxes are matched with the set of subject detection boxes. If the matching result satisfies the above-mentioned spatial positional relationship constraints, the low-confidence face detection boxes are recalled to the high-confidence face detection boxes, thereby improving the accuracy of face detection box prediction, improving the accuracy of subsequent face optimization algorithms, and improving the user's photo-taking experience.

[0026] According to the first aspect, or any implementation of the first aspect above, the step of matching each face detection box in the low-confidence face detection box set with each subject detection box in the subject detection box set includes: if the current second face detection box in the low-confidence face detection box set is inside any subject detection box in the subject detection box set, and the current second face detection box is located in the upper half region of the subject detection box, and the width of the current second face detection box is less than 1 / 2 of the subject detection box, then the matching is determined to be successful.

[0027] According to the first aspect, or any implementation of the first aspect above, when the face detection model supports the subject detection task, the subject detection model is a submodule of the face detection model.

[0028] According to the first aspect, or any implementation of the first aspect above, before clustering each face detection box in the high-confidence face detection box set to obtain at least one grouping line and its corresponding second spatial coordinates, the method further includes: acquiring at least one historical frame image; predicting face detection boxes in the historical frame image based on the face detection model to obtain a historical face detection box set corresponding to the historical frame image; the information of each face detection box in the historical face detection box set includes confidence and third spatial coordinates; matching each face detection box in the historical face detection box set and the low-confidence face detection box set based on the third spatial coordinates and the first spatial coordinates; if the matching is successful, acquiring the corresponding historical face detection box; if the confidence of the historical face detection box is greater than the first preset confidence threshold, recalling the corresponding face detection box to the high-confidence face detection box set.

[0029] Understandably, in photography scenarios, photographers typically take multiple shots of the same scene and select the best image as the final image. Furthermore, because the intervals between shots are short, the range of movement of the same subject's face is relatively small. Therefore, the face detection bounding boxes of the same subject in multiple historical frames can be matched with the face detection bounding boxes in the current frame to determine whether the face detection bounding box in the current frame represents a missed real face.

[0030] For example, such as Figure 9 The two frames shown in (1) interface 50a and (2) interface 50b are as follows: the image in interface 50a can be identified as a historical frame image, and the image in interface 50b can be identified as the current frame image.

[0031] Therefore, in this embodiment of the application, before clustering high-confidence face detection boxes to obtain the group photo line, high-confidence face detection boxes in historical frame images are matched with low-confidence face detection boxes in the current frame image, thereby recalling low-confidence face detection boxes, which improves the accuracy of face detection box prediction, improves the accuracy of subsequent face optimization algorithms, and enhances the user's photo-taking experience.

[0032] According to the first aspect, or any implementation of the first aspect above, the step of matching each face detection box in the historical face detection box set and the low-confidence face detection box set based on the third spatial coordinates and the first spatial coordinates includes: calculating the overlap between the current historical face detection box in the historical face detection box set and the current face detection box in the low-confidence face detection box set based on the overlap between the third spatial coordinates and the first spatial coordinates; and determining that the current historical face detection box and the current face detection box are successfully matched if the overlap is greater than a preset overlap threshold.

[0033] For example, the above overlap calculation can be based on Figure 8 The calculation method shown above is used. The preset overlap threshold can be set to 0.5.

[0034] According to the first aspect, or any implementation of the first aspect above, after recalling the corresponding face detection box to the high-confidence face detection box set when the comparison result satisfies the spatial positional relationship constraint of the first group photo line, the method further includes: performing non-maximum suppression on the face detection boxes in the high-confidence face detection box set to obtain the final face detection result.

[0035] Secondly, embodiments of this application provide an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform the methods of the first aspect or any possible implementation thereof.

[0036] Thirdly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0037] Fourthly, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0038] Fifthly, embodiments of this application provide a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals. Attached Figure Description

[0039] Figure 1 A schematic diagram of the hardware structure of an electronic device as an example;

[0040] Figure 2 A schematic diagram of the software structure of an electronic device as an example;

[0041] Figure 3 This is a user operation diagram shown as an example;

[0042] Figure 4 This is a schematic diagram of the first group photo scene as an example;

[0043] Figure 5 This is a schematic diagram of a second group photo scene as an example;

[0044] Figure 6 This is an example illustration of matching a face bounding box with a subject bounding box;

[0045] Figure 7 This is an example of a schematic diagram showing the interface of the face bounding box and the subject bounding box matching results;

[0046] Figure 8 This is an example of an overlap calculation diagram;

[0047] Figure 9 This is a schematic diagram of the interface for the overlap calculation results, as shown as an example.

[0048] Figure 10 This is a schematic diagram of a group photo line matching interface as an example.

[0049] Figure 11 This is a schematic diagram of the first process of face detection as an example.

[0050] Figure 12 This is a schematic diagram of the second process of face detection as an example;

[0051] Figure 13 This is a schematic diagram of the third process of face detection, as an example. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0054] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0057] To better understand the technical solutions provided in the embodiments of this application, before describing the technical solutions of the embodiments of this application, the hardware structure of the terminal devices (such as mobile phones, tablets, smart wearable devices, and other terminal devices with image acquisition functions) to which the embodiments of this application are applicable will first be described with reference to the accompanying drawings. For ease of explanation, Figure 1 Let's take a mobile phone as an example.

[0058] See Figure 1 The mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0059] The processor 110 may include one or more processing units, such as an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc., which will not be listed here and this application does not limit them.

[0060] The controller mentioned above, which serves as the processing unit, can be the central nervous system and command center of the mobile phone 100. In practical applications, the controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0061] The aforementioned modulation and demodulation processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal and transmits the demodulated low-frequency baseband signal to the baseband processor for processing.

[0062] The aforementioned baseband processor is used to process the low-frequency baseband signal transmitted by the regulator and then transmit the processed low-frequency baseband signal to the application processor.

[0063] It should be noted that in some implementations, the baseband processor can be integrated into the modem, meaning the modem can have the functionality of a baseband processor.

[0064] The aforementioned application processor is used to output sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or to display images or videos through display screen 194.

[0065] Regarding the aforementioned neural network processor, specifically in the technical solutions provided in this application embodiment, it can be used to train the neural network model for recognizing faces in images as described in this application embodiment. Understandably, to reduce resource consumption on the mobile phone 100, the neural network model can also be trained by a cloud server or other server and then distributed to the mobile phone 100.

[0066] The aforementioned video codecs are used for compressing or decompressing digital video. For example, mobile phone 100 may support one or more video codecs. Thus, mobile phone 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0067] The aforementioned ISP is used to output digital image signals to DSP for processing.

[0068] The DSP mentioned above is used to convert digital image signals into standard RGB, YUV, and other image signal formats.

[0069] Furthermore, it should be noted that, regarding the processor 110 including the aforementioned processing units, in some implementations, the different processing units can be independent devices. That is, each processing unit can be considered as a processor. In other implementations, the different processing units can also be integrated into one or more processors. For example, in some implementations, the modem processor can be an independent device. In other implementations, the modem processor can be independent of the processor 110 and housed in the same device as the mobile communication module 150 or other functional modules.

[0070] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0071] In addition, the processor 110 may also include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., which will not be listed here, and this application does not impose any limitations on them.

[0072] In addition, processor 110 may also include memory for storing instructions and data. In some implementations, the memory in processor 110 is a cache memory. This memory can store instructions or data that processor 110 has just used or is recurring. If processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of processor 110, and thus improves system efficiency.

[0073] In this embodiment of the application, the processor 110 can be used to acquire the video stream being captured by the user from the camera, and to predict the face detection box for each frame of the image based on the acquired video stream.

[0074] See also Figure 1 The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0075] See also Figure 1 The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area.

[0076] See also Figure 1 The charging management module 140 is used to receive charging input from the charger.

[0077] See also Figure 1 The wireless communication function of mobile phone 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0078] It should be noted that in some implementations, the neural network model used for face detection can be trained by a cloud server or other server. In this scenario, mobile phone 100 can communicate with the cloud server or other server providing the neural network via mobile communication module 150 or wireless communication module 160. For example, mobile phone 100 can send a request to the cloud server to obtain or update the neural network model via mobile communication module 150. Correspondingly, the cloud server can send the trained neural network model to mobile phone 100 based on the request.

[0079] In addition, it should be noted that in scenarios where the neural network model is trained by a cloud server (or other server), the cloud server can customize the neural network model to suit the different configurations of the mobile phone 100 according to their respective customization requirements, and update and iterate the training of the neural network model based on the face detection results fed back by different mobile phones 100.

[0080] See also Figure 1 The audio module 170 may include a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, etc. For example, the mobile phone 100 can implement audio functions through the application processor and the speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, etc. in the audio module 170. Examples include recording and video recording functions.

[0081] See also Figure 1 The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc., which will not be listed here, and this application does not limit them.

[0082] See also Figure 1 The buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The mobile phone 100 can receive button input and generate button signal inputs related to the user settings and function control of the mobile phone 100.

[0083] See also Figure 1 Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0084] See also Figure 1 The indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0085] See also Figure 1 The camera 193 is used to capture still images or videos. The mobile phone 100 can achieve its shooting function through an ISP, camera 193, video codec, GPU, display 194, and application processor. Specifically, an object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some implementations, the mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0086] See also Figure 1 The display screen 194 is used to display images, videos, etc.

[0087] That concludes the introduction to the hardware structure of the Mobile 100. It should be understood that... Figure 1 The mobile phone 100 shown is just an example. In a specific implementation, the mobile phone 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0088] To better understand Figure 1 The software structure of the mobile phone 100 shown is described below. Before describing the software structure of the mobile phone 100, the possible architectures for the software system of the mobile phone 100 will be explained first.

[0089] Specifically, in practical applications, the software system of Mobile 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.

[0090] Furthermore, it is understood that the software systems currently used by mainstream terminal devices include, but are not limited to, Windows, Android, and iOS systems. For ease of explanation, this application embodiment uses the layered architecture of the Android system as an example to illustrate the software structure of the mobile phone 100.

[0091] Furthermore, the face detection scheme provided in the embodiments of this application is also applicable to other systems in specific implementations.

[0092] See Figure 2 This is a software structure block diagram of the mobile phone 100 according to an embodiment of this application.

[0093] like Figure 2 As shown, the layered architecture of the mobile phone 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some implementations, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0094] The application layer can include a series of application packages. For example... Figure 2 As shown, the application package may include applications such as camera, recorder, settings, Bluetooth, gallery, etc., which will not be listed here, and this application does not impose any restrictions on them.

[0095] Specifically, in the embodiments of this application, the entry point for the face detection scheme can be set in the camera application, for example... Figure 3 Control 10b-1 in (2). Similarly, after a third-party application calls the interface provided by the face detection scheme in this application embodiment, its implementation entry point can also be set in the third-party application.

[0096] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. In some implementations, these APIs and frameworks can be described as functions.

[0097] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0098] The system library and runtime layer includes the system libraries and the Android Runtime. The Android Runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0099] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0100] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0101] The system library can include multiple functional modules. For example: surface manager, media libraries, face recognition engine, 2D graphics engine (e.g., SGL), etc.

[0102] Specifically, in this embodiment, the face recognition engine is pre-loaded with a pre-trained model, including a deep learning model for face recognition. The camera application inputs a video stream or image into the face recognition engine, which then calls the corresponding processing model to process the video stream or image, thereby outputting the final face recognition result.

[0103] Understandably, the 2D graphics engine mentioned above is a 2D drawing engine.

[0104] The HAL layer is the interface layer located between the operating system kernel and the hardware circuitry. The HAL layer includes, but is not limited to: the camera hardware abstraction layer, the display driver hardware abstraction layer, and the power supply hardware abstraction layer.

[0105] The kernel layer is the layer between hardware and software. It contains at least display drivers, camera drivers, power management drivers, and sensor drivers. The display driver can include drivers for the secondary screen and the main screen. The sensor drivers can include drivers for gyroscope sensors, accelerometers, Hall effect sensors, etc., used to detect angle changes between the main and secondary screens of the foldable phone. The camera driver powers the corresponding camera. The capacitor management driver powers the power management module to supply power to the display.

[0106] It should be understood that the above description is merely an example provided to better understand the technical solution of this embodiment, and is not intended as the only limitation on this embodiment.

[0107] Furthermore, it is understood that the above division of functional modules is merely an example provided to better understand the technical solution of this embodiment, and is not intended to be the sole limitation of this embodiment. In practical applications, the above functions can also be integrated into a single functional module, and this embodiment does not impose any restrictions on this.

[0108] That concludes the introduction to the software structure of the mobile phone 100. As you can understand, Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0109] Group activities are an essential part of daily life, such as group learning, birthday parties, and family gatherings. During these activities, people often take group photos to commemorate the occasion. When taking group photos, people usually arrange themselves in order of height so that everyone can be clearly and completely represented in the picture.

[0110] For example, after a group learning session, people usually take a group photo at the learning location. During the photo session, the group is usually divided into two rows, one in front and one behind. To further prevent those in the front row from blocking those in the back row, those in the front row may be asked to squat, or those in the back row may be asked to stand on stools.

[0111] However, during the photo shoot, due to the large number of people in the group photo and the possibility of posture adjustments by those taking the picture, it's impossible for every face to be clearly and completely presented in the final photo, thus reducing the user's photography experience. Therefore, mobile phone manufacturers also provide users with camera algorithms and models to optimize poor-quality faces and images.

[0112] To better understand the process of capturing group photos with a mobile phone, the following is combined with the attached... Figure 3 To be continued Figure 5 Please provide an explanation.

[0113] See Figure 3 In Figure (1), a mobile phone interface 10a is shown as an example. For example, the interface 10a displays icons of multiple applications, such as the camera application icon 10a-1, and icons of applications such as contacts, phone, messages, clock, calendar, gallery, memo, file manager, email, music, calculator, video, recorder, weather, browser, settings, etc.

[0114] It should be noted that, among some possible implementation methods, Figure 3 The interface 10a shown in (1) can be called the main interface. When the user clicks the icon 10a-1 in the interface 10a, the user can use the camera application to perform shooting functions, such as taking photos, shooting (recording) videos, etc.

[0115] Continue to refer to Figure 3In example (1), when a user clicks the camera application icon 10a-1, the phone responds to the user's operation, recognizes the icon corresponding to the user's click as the camera application icon, and then calls the corresponding interface in the application framework layer to start the camera application. It also calls the kernel layer to start the camera driver and captures an image stream (in this case, a preview stream) through the camera. At this time, the phone displays the camera application interface, for example... Figure 3 The interface 10b shown in (2) will display the screen corresponding to the preview stream, such as Figure 3 The image shown in (2) is a group photo of the group.

[0116] As mobile phone camera functions have improved, camera apps support an increasing number of shooting modes. For example, shooting modes may include aperture mode, night mode, portrait mode, photo mode, video mode, smile mode, and professional mode, etc. (See also...) Figure 3 The mode options are shown in the shooting mode list 10b-1 in (2).

[0117] For example, when a user taps the icon for a specific shooting mode, the phone displays the camera app interface for that mode. For instance, if the user taps the portrait mode icon, the phone displays the camera app interface for portrait mode, which can be referenced... Figure 3 Interface 10b is shown in (2).

[0118] It should be noted that in some implementations, after the phone launches the camera app, the camera app defaults to portrait mode, meaning portrait mode is the default shooting mode for the camera app. In portrait mode, a shutter control is also displayed on interface 10b. In some implementations, the user can take a picture by clicking the shutter control once or multiple times. For example, when the user clicks the shutter control, such as twice, the phone detects the user action on the shutter control and responds by taking a picture, resulting in an image like... Figure 4 The group photo interface 20a shown in (1) and Figure 5 The group photo interface 30a shown in (1) is shown in the middle.

[0119] Meanwhile, in portrait mode, to provide users with a better photography experience, mobile phone manufacturers use built-in face detection models (also known as face recognition models) to obtain face detection boxes that need optimization in the image, and then use built-in image algorithms to optimize the imaging effect of the faces in the obtained face detection boxes. However, the recall rate of the face detection model is strongly tied to the size and quality of the faces in the input image. In group photos, the faces in the input image are small in scale and of poor quality (blurred, occluded), resulting in a low face recall rate, which seriously reduces the shooting effect in group photos.

[0120] For example, such as Figure 4 In the group photo interface shown in interface 20a (1), the participants are assigned to two rows. The distance between people in the first row is appropriate, while the distance between people in the second row is crowded. Among them, the faces of people numbered RW2 and RW3 in interface 20a are relatively small, which will lead to the risk of face detection failure for people RW2 and RW3, thereby reducing the optimization effect of the image algorithm.

[0121] See also Figure 4 (2) Figure 4 (2) illustrates an example of a face detection diagram. In this diagram, the square indicates that the selected face is a high-confidence face detection box, which can be processed normally (e.g., skin smoothing, sharpening, etc.) when using the image algorithm provided by the mobile phone manufacturer; the elliptical box indicates that the selected face is a low-confidence face detection box (e.g., the face detection boxes of person RW2 and person RW3), which will be ignored in the subsequent algorithm processing, resulting in a reduction in image optimization effect.

[0122] For example, such as Figure 5 In the group photo interface shown in interface 30a (1), the distance between people in the first row is appropriate, while the distance between people in the second row is crowded. Among them, people numbered RW2 and RW3 in interface 20a have small faces; at the same time, because person RW1 in the first row has a side profile, this will lead to the risk of face detection failure for people RW1, RW2 and RW3, thereby reducing the optimization effect of the image algorithm.

[0123] See also Figure 5 (2) Figure 5 (2) illustrates another face detection diagram. In this diagram, the square indicates that the selected face is a high-confidence face detection box, which can be processed normally (e.g., skin smoothing, sharpening, etc.) when using the image algorithm provided by the mobile phone manufacturer; the elliptical box indicates that the selected face is a low-confidence face detection box (e.g., the face detection boxes of person RW1, person RW2, and person RW3), which will be ignored in the subsequent algorithm processing, resulting in a reduction in image optimization effect.

[0124] Understandably, before image algorithms process the images of people in a group photo, they need to first use a model to detect faces, and then set the obtained high-confidence face detection boxes as the face detection boxes to be processed, thereby obtaining optimized results through algorithmic processing. However, as mentioned above... Figure 4 Personnel RW2 and Personnel RW3, and Figure 5Persons RW1, RW2, and RW3 were identified as low-confidence face detection bounding boxes, preventing the image algorithm from optimizing them and resulting in poor image quality in the final output. Figure 4 Personnel RW2 and Personnel RW3, and Figure 5 The shooting results of personnel RW1, RW2, and RW3 were significantly worse than those of other personnel.

[0125] One implementation approach is to improve face recall in the face detection model by introducing additional computation (e.g., increasing model size or input image size), thereby improving shooting results. However, this undoubtedly increases the performance requirements of the phone and also increases its power consumption.

[0126] In one implementation, the face recall rate in the face detection model can be improved by increasing the false face detection rate. However, since there may be some non-face targets in target recognition, increasing the false face detection rate also increases the number of non-face targets in the target recognition result. Applying face enhancement effects (such as beauty filters or FaceSR) to non-face targets will undoubtedly lead to a deterioration in the final image quality.

[0127] In view of this, embodiments of this application provide a face detection method that improves the face recall rate in the face detection model and enhances the user's photo-taking experience without increasing the performance requirements or power consumption of the mobile phone.

[0128] In one implementation, the image is still taken using a camera app. Figure 5 Taking the group photo interface shown in (1) as an example, based on the face detection method provided in the embodiments of this application, based on condition ①: the relationship between the spatial position and size of the face detection boxes among multiple group photo subjects (people in the image) in the current image frame, condition ②: the relationship between the spatial position and size of the face detection box of the same subject and the subject box in the current image frame, and condition ③: the relationship between the spatial position and size of the face detection box of the same subject between the current frame and the historical frame, the above conditions ①, ② and ③ are applied to the face detection of the multi-person group photo scene of the face detection model to improve the face detection recall rate, thereby recalling the filtered low-confidence face detection boxes without introducing additional calculations and increasing the false detection rate of faces, and improving the user's photo-taking experience.

[0129] To explain this beneficial effect in detail, the following is combined with... Figures 6 to 10 Please provide an explanation.

[0130] See Figure 5In section (1), an exemplary group photo interface 30a is shown. Exemplarily, the people taking photos on this interface 30a are arranged in two rows. Before processing the image using image algorithms, a deep learning model (i.e., a face detection model) is used to infer the spatial positions of face detection boxes and subject (i.e., human body) detection boxes on the image, resulting in the following... Figure 5 Interface 30b is shown in (2).

[0131] It is understood that the aforementioned deep learning models now have mature model architectures and datasets. In this application embodiment, it is only necessary to select a suitable model and dataset, learn and test to meet the standards, and then use it to predict face bounding boxes and subject bounding boxes for the group photo image shown in the group photo interface 30a. This application embodiment does not impose any restrictions on this.

[0132] For example, the deep learning model described above can use open-source models on the MMdetection platform to predict face bounding boxes and subject bounding boxes.

[0133] See Figure 5 (2) illustrates all the face bounding boxes in the group photo image. Among them, the face bounding boxes of person RW1, person RW2 and person RW3 are low-confidence face bounding boxes and need to be recalled.

[0134] Understandably, in face detection tasks in group photos, the shooting distance is far and the number of pixels occupied by the face in the image is small, resulting in less feature information available to the face detection model for the face region. Therefore, the confidence of the face detection model when making predictions is low, leading to low recall.

[0135] In one implementation, in a scenario where the face detection recall rate is improved based on condition ① above, the obtained subject bounding box can be used as prior information for recalling low-confidence face bounding boxes. By comparing the spatial position and size relationship between the low-confidence face bounding box and the subject bounding box, it can be determined whether the current low-confidence face bounding box is a missed face bounding box.

[0136] For example, see Figure 6 , Figure 6 An exemplary diagram illustrating the matching of a face bounding box and a subject bounding box is shown. Figure 6 In the diagram, the face bounding box is RNK1, and the main body bounding box is ZTK1.

[0137] Understandably, in practice, there are certain spatial and size constraints between the face frame and the main frame. For example, the face frame should be inside the main frame, and it should be located in the upper half of the main frame, while the width of the face frame should be less than half the width of the main frame.

[0138] For example, the above constraints are applied to Figure 5 When recalling low-confidence face frames in interface 30b shown in (2), the following is displayed: Figure 7 The interface shown is 40b.

[0139] Understandably, since person RW1 is in the front row, their corresponding subject bounding box can be accurately predicted. However, people RW2 and RW3 are in the back row, and their bodies are obscured by people on either side, so their subject bounding boxes cannot be accurately predicted. Therefore, the above constraints can be applied to the face bounding box recall of person RW1.

[0140] Meanwhile, when the above constraints are applied to the face frame recall of person RW1, the corresponding requirements are met, namely, the face frame is inside the main frame and the face frame is in the upper half of the main frame, while the width of the face frame is less than 1 / 2 of the width of the main frame.

[0141] Based on the above conditions, the face bounding box corresponding to person RW1 in interface 40b can be determined as a missed face and recalled. This improves the recall rate of the face recognition model without increasing computational load or false positive rate, thereby enhancing the user's photo-taking experience.

[0142] In another implementation, in the scenario of improving face detection recall based on condition ② above, multiple historical images can be continuously acquired, and the spatial positions of face bounding boxes of the same subject in the multiple historical images and the current frame image can be associated. Therefore, based on the confidence level of the face bounding boxes in the historical frames, it can be determined whether the current low-confidence face bounding box is a missed face bounding box.

[0143] Understandably, in images captured in multiple consecutive frames, the short intervals between frames result in a small range of facial movement for the same subject. Therefore, the spatial position of face bounding boxes within these frames is subject to certain constraints. For example, when calculating the Intersection over Union (IOU) for the same subject in two consecutive frames, if the IOU exceeds a certain preset threshold, the subject in the two consecutive frames can be determined to be the same subject. Furthermore, if the face bounding box of the subject in the previous frame is a high-confidence face bounding box, the face bounding box in the current frame can also be considered a high-confidence face bounding box.

[0144] For an example, see the method for calculating overlap. Figure 8 The Intersection over Union (IOU) is obtained by finding the intersection and union of the face bounding boxes in two images, and then dividing the intersection by the union. When the IOU exceeds a certain preset threshold, it can be determined that the subjects in the two images are the same subject.

[0145] It is understandable that the aforementioned preset overlap threshold can be set based on the frame rate when the camera takes multiple frames of photos.

[0146] For example, when the camera's frame rate is 30 frames per second (FPS), the overlap threshold can be set to 0.7. When the camera's frame rate is 6 frames per second (FPS), the overlap threshold can be set to 0.45.

[0147] For example, see Figure 9 Examples (1) and (2) illustrate two frames of the same group photo scene, one before the other. Figure 9 In interface 50a shown in (1), the face bounding box angle of person RW1 is normal, and the face bounding box obtained by the face recognition model is a high-confidence face bounding box. Figure 9 In the interface 50b shown in (2), the angle of the face frame of person RW1 is offset. At this time, the face frame obtained by the face recognition model is a low-confidence face frame.

[0148] By calculating the overlap between the face bounding boxes in interface 50a and interface 50b, and comparing the obtained overlap with the overlap threshold in this scene, it can be determined that although the face bounding box of person RW1 in interface 50b is a low-confidence face bounding box, it is a missed face and can be recalled. This improves the recall rate of the face recognition model without increasing computational load or false positive rate, thereby enhancing the user's photo-taking experience.

[0149] In another implementation, in scenarios where face detection recall is improved based on condition ③ above, the positional relationships of faces near the group photo line can be controlled using the group photo line. This allows for the recall of low-confidence face bounding boxes near the group photo line.

[0150] Understandably, in group photos, the typical characteristic is that the subjects are arranged in rows. Therefore, the row vector of the subjects can be called the group photo line. The group photo line can be obtained by clustering the face bounding boxes of each subject. All subjects in the group photo should have their corresponding face bounding boxes distributed near the group photo line.

[0151] For example, see Figure 10 , Figure 10 Interface 60a exemplarily illustrates two group photo lines in a group photo scene. In this interface, by clustering the y-coordinates of each high-confidence face bounding box, the cluster centers are the y-coordinates of each group photo line. For example... Figure 10 The center line HYX1 and line HYX2 represent the two group photo lines in this group photo scene.

[0152] It should be noted that the implementation of the clustering algorithm can be customized according to the group photo scenario. For example, in this embodiment, the k-means clustering algorithm is used to calculate the group photo line, and the Elbow Method is used to evaluate the optimal number of group photo lines.

[0153] After obtaining the group photo lines, calculate the average width (w~) and height (h~) of the high-confidence face bounding boxes on each group photo line. Based on the y-coordinate of each group photo line and the average width and height of its corresponding high-confidence face bounding boxes, determine the spatial position and size relationship of the low-confidence face bounding boxes. If the low-confidence face bounding box meets the preset spatial positional relationship constraints of the group photo lines, it can be determined that the low-confidence face bounding box is a missed frame face and it can be recalled.

[0154] It is understandable that the aforementioned spatial positional constraints of the group photo line can be customized according to different group photo scenarios. For example, the constraints can be set such that the low-confidence face frame must fall on the group photo line, and its width and height must differ from the average width and average height of the face frame on the current group photo line by a preset threshold k%.

[0155] The k-value can be calculated by removing the largest and smallest face bounding boxes in the current image frame, then calculating the average width and height of the high-confidence face bounding boxes on each group photo line. The width of the largest face bounding box is then compared to the average width on each group photo line to calculate a first ratio, the height of the largest face bounding box to calculate a second ratio, the width of the smallest face bounding box to calculate a third ratio, and the height of the smallest face bounding box to calculate a fourth ratio. The larger of the first and third ratios is set as a preset threshold k% for width, and the larger of the second and fourth ratios is set as a preset threshold k% for height.

[0156] By calculating whether each low-confidence face frame satisfies the aforementioned preset spatial positional relationship constraints of the group photo line, it can be determined whether it needs to be recalled.

[0157] For example, in interface 60a, the face frames of persons RW1, RW2, and RW3 all satisfy the aforementioned preset spatial positional relationship constraints of the group photo line. Therefore, it is necessary to recall their corresponding face frames. This improves the recall rate of the face recognition model without increasing the computational load or the false detection rate, thereby enhancing the user's photo-taking experience.

[0158] It should be noted that conditions ①, ② and ③ above can be applied to the face detection task of the face detection model in any combination to improve the recall rate of the face detection task. This application embodiment does not limit this.

[0159] To better understand the process of recalling low-confidence faces using the face detection method provided in this application, the following embodiments still use mobile phone 100 as an example to illustrate the process of recalling low-confidence faces in the face detection task.

[0160] For example, such as Figure 11 As shown, in one implementation, the face detection method includes the following steps:

[0161] S101. The application is started in response to the first operation command triggered by the user.

[0162] It is understandable that the aforementioned first operation command can be a user's command to launch the camera application on the phone in any way. For example, by tapping the camera application shortcut on the touchscreen, or by triggering the camera application launch command through predefined shortcut gestures and shortcut keys.

[0163] For example, such as Figure 3 As shown in Figure (1), the user clicks icon 10a-1 to trigger the camera application launch command.

[0164] After receiving the command, the phone launches the camera app and displays the following: Figure 3 The preview interface shown in (2) is shown in the middle.

[0165] S102. Acquire the image stream in response to a second operation command triggered by the user.

[0166] It is understood that the aforementioned image stream can be a preview stream obtained by the camera application through the camera lens before the user clicks the shutter control of the camera application; it can also be a video stream obtained after the user clicks the shutter control of the camera application once; or it can be a video stream obtained after the user clicks the shutter control of the camera application multiple times. This application embodiment does not impose any limitations on this.

[0167] For example, the image stream described above may contain one or more frames of images. For example Figure 4 The image shown in (1) and Figure 5 The image shown in (1) is shown in the middle.

[0168] S103. Obtain the current frame image.

[0169] It is understood that the aforementioned current frame image can be an image acquired by the camera from the aforementioned image stream in chronological order; or it can be an image acquired from the aforementioned video stream based on an image algorithm. This application embodiment does not impose any limitations on this.

[0170] S104. Perform face detection box prediction on the current frame image to obtain a set of face detection boxes.

[0171] It is understandable that the above steps for predicting face detection bounding boxes in the current frame image can be based on a pre-trained face recognition model. Currently, mature face detection model frameworks and corresponding datasets are available. In this embodiment, a suitable model and dataset can be selected based on the actual scenario requirements, and prior learning and testing are sufficient.

[0172] For example, the face detection model in this application embodiment may be an open-source model using object detection (MMdetection).

[0173] Furthermore, the face detection model described above can also use YOLO series algorithms or R-CNN series algorithms, and this application embodiment does not limit this.

[0174] Specifically, when predicting face detection bounding boxes for the current frame image using a face detection model, the face detection model outputs the confidence score, width, height, and spatial coordinates of each face detection bounding box.

[0175] It is understandable that the above spatial coordinates refer to the spatial position in a coordinate system with the top left corner of the image as the origin and a single pixel as the unit length.

[0176] S105. The face detection box set is divided based on the first confidence threshold and the second confidence threshold to obtain a high-confidence face detection box set and a low-confidence face detection box set.

[0177] It is understood that the aforementioned first confidence threshold and second confidence threshold are used to divide the aforementioned set of face detection boxes into a set of high-confidence face detection boxes and a set of low-confidence face detection boxes. The aforementioned first confidence threshold is greater than the aforementioned second confidence threshold, and the aforementioned first confidence threshold and the aforementioned second confidence threshold are combined to form the low-confidence threshold range.

[0178] For example, the first confidence threshold can be set to 0.5, and the second confidence threshold can be set to 0.3. When the confidence of a face detection bounding box is greater than or equal to 0.5, the face detection bounding box is classified into the high-confidence face detection bounding box set; when the confidence of a face detection bounding box is greater than or equal to 0.3 and less than 0.5, the face detection bounding box is classified into the low-confidence face detection bounding box set. When the confidence of a face detection bounding box is less than 0.3, the face detection bounding box is directly discarded.

[0179] Understandably, in face detection tasks within group photos, the shooting distance is relatively far, and the number of pixels occupied by faces in the image is small. Face detection models have limited available feature information for face regions, resulting in low confidence in predicting small faces and consequently low recall. Therefore, each face detection box in the set of low-confidence boxes may be a missed face detection box, and once it is determined to be a missed face detection box, it needs to be recalled.

[0180] For example, such as Figure 4 The face detection bounding boxes corresponding to personnel RW2 and personnel RW3 shown in (2) are low-confidence face detection bounding boxes, and Figure 5 The face detection boxes corresponding to personnel RW1, personnel RW2 and personnel RW3 shown in (2) are low confidence face detection boxes.

[0181] S106. Cluster the face detection boxes in the high-confidence face detection box set to obtain the average width and height of the face detection boxes of all face boxes on the group photo line.

[0182] It is understandable that in group photos, the typical arrangement of the main subjects (people) is in rows. Therefore, in this embodiment, the row vector of the main subjects in the group photo is defined as the group photo line. For example... Figure 4 and Figure 5 In the group photo scene, the main subjects are arranged in two rows.

[0183] For example, such as Figure 10 Lines HYX1 and HYX2 in the diagram represent the two lines in the group photo scene.

[0184] Specifically, by clustering all face detection boxes in the high-confidence face detection box set, the group photo line in this scene can be obtained.

[0185] Furthermore, the clustering algorithm described above can be any algorithm capable of clustering face detection boxes, including but not limited to k-means algorithm, support vector machine (SVM) or deep learning model.

[0186] For example, the clustering algorithm described above can use the k-means algorithm to calculate the group line and use the Elbow Method to evaluate the optimal number of group lines.

[0187] Specifically, after obtaining the aforementioned group photo lines, the average width and average height of the face detection boxes in the cluster corresponding to each group photo line are calculated. These clusters represent all face detection boxes that the group photo lines pass through.

[0188] S107. Based on the group photo line and its corresponding average width and height, recall the face detection boxes in the low-confidence face detection box set to the high-confidence face detection box set.

[0189] Specifically, if the group photo line passes through a face detection box in the set of low-confidence face detection boxes, and the difference between the width and height of the face detection box and the average width and average height of the face detection boxes in the current group photo line is within a preset threshold k%, then the face detection box can be determined as a missed real face detection box and needs to be recalled.

[0190] Furthermore, if the spatial coordinates of any group photo line fall within the spatial coordinates of the current face detection box, and it is determined that the group photo line passes through the current face detection box, and the difference between the width of the current face detection box and the average width of the group photo line is within the preset threshold k% range corresponding to the group photo line, and the difference between the height of the current face detection box and the average height of the group photo line is within the preset threshold k%, then the current low-confidence face detection box can be identified as a real face detection box that can be recalled.

[0191] The k-value can be calculated by removing the largest and smallest face bounding boxes in the current image frame, then calculating the average width and height of the high-confidence face bounding boxes on each grouping line. The width of the largest face bounding box is then compared to the average width on each grouping line to calculate a first ratio, the height of the largest face bounding box to calculate a second ratio, the width of the smallest face bounding box to calculate a third ratio, and the height of the smallest face bounding box to calculate a fourth ratio. The larger of the first and third ratios is set as a preset threshold k1% for the width, and the larger of the second and fourth ratios is set as a preset threshold k2% for the height.

[0192] In one implementation, the aforementioned preset threshold k1% and the aforementioned preset threshold k2% can be further maximized, thereby setting the maximum value (k1% or k2%) as the aforementioned preset threshold k.

[0193] For example, in Figure 10In interface 60a, the face detection bounding boxes for persons RW1, RW2, and RW3, after the above judgment, are all real face detection boxes that were missed. Therefore, it is necessary to recall their corresponding face boxes. This improves the recall rate of the face recognition model without increasing the computational load or the false detection rate, thereby improving the user's photo-taking experience.

[0194] S108. Perform nonmaximum suppression on the high-confidence face detection bounding boxes to obtain the face detection results.

[0195] Understandably, group photos include not only the subject but also background content, which may lead to the detection of redundant bounding boxes. Therefore, it is necessary to remove these redundant bounding boxes to retain only the most likely faces.

[0196] Specifically, in the high-confidence face detection box set, all face detection boxes with the same spatial coordinates are obtained. These face detection boxes are sorted based on their confidence level, and the face detection box with the highest confidence level is selected as the face detection box corresponding to that spatial coordinate. Other face detection boxes are deleted, and the high-confidence face detection box set is updated. The overlap between each pair of face detection boxes in the updated high-confidence face detection box set is calculated. If the overlap exceeds a preset overlap threshold, one of the face detection boxes is deleted. The final high-confidence face detection box set is output as the face detection result. The overlap threshold can be set to 0.5.

[0197] S109. Send the face detection results to the face optimization module.

[0198] The face detection results are sent to the face optimization module so that the face optimization module can optimize the faces in each face detection frame based on the preset image algorithm.

[0199] Therefore, this application embodiment obtains a group photo line by clustering the face detection boxes in the high-confidence face detection box set, and then recalls the face detection boxes in the low-confidence face detection box set based on the group photo line. This recalls the filtered low-confidence face boxes without introducing additional computation or increasing false detections, thereby improving the user's photo-taking experience.

[0200] In one implementation, the pre-built face detection model in the mobile phone simultaneously supports both face detection bounding box prediction and subject detection bounding box prediction tasks. That is, the face detection model includes a sub-module face detection model and a sub-module subject detection model. This face detection method includes the following steps:

[0201] S101. The application is started in response to the first operation command triggered by the user.

[0202] S102. Acquire the image stream in response to a second operation command triggered by the user.

[0203] S103. Obtain the current frame image.

[0204] S104. Perform face detection box prediction on the current frame image to obtain a set of face detection boxes.

[0205] S105. The face detection box set is divided based on the first confidence threshold and the second confidence threshold to obtain a high-confidence face detection box set and a low-confidence face detection box set.

[0206] S201. Predict the subject detection box for the current frame image to obtain a set of subject detection boxes.

[0207] Specifically, the subject detection model in the face detection model is used to predict subject detection boxes for the current frame image, resulting in a set of subject detection boxes. Each subject detection box in this set includes spatial coordinates, width, and height.

[0208] It should be noted that if the face detection model does not support the subject detection task and the mobile phone does not have a pre-installed subject detection model, the mobile phone can skip steps S201 and S202.

[0209] S202. Match each face detection box in the low-confidence face detection box set with each subject detection box in the subject detection box set. If a match is successful, recall it to the high-confidence face detection box set.

[0210] It is understandable that when a face detection bounding box is a true missed face, there is a spatial relationship constraint between the face detection bounding box and its corresponding subject detection bounding box. Therefore, the subject detection bounding box can be used as prior information for recalling low-confidence face detection bounding boxes. When the low-confidence face detection bounding box and any subject detection bounding box in the set of subject detection bounding boxes have this spatial relationship constraint, the face detection bounding box is determined to be a missed face detection bounding box.

[0211] For example, the spatial relationship is constrained such that the face detection bounding box is inside the main detection bounding box, and the face detection bounding box is located in the upper half of the main detection bounding box, and the width of the face detection bounding box is less than half the width of the main detection bounding box. For example, as shown... Figure 6 The diagram shows the spatial relationship between the face detection bounding box and the subject detection bounding box.

[0212] For example, in Figure 7 By comparing the face detection bounding box with the subject detection bounding box, it can be determined that the face detection bounding box corresponding to person RW1 is a real face bounding box that was missed and needs to be recalled to the high-confidence face detection bounding box.

[0213] S106. Cluster the face detection boxes in the high-confidence face detection box set to obtain the average width and height of the face detection boxes of all face boxes on the group photo line.

[0214] S107. Based on the group photo line and its corresponding average width and height, recall the face detection boxes in the low-confidence face detection box set to the high-confidence face detection box set.

[0215] S108. Perform nonmaximum suppression on the high-confidence face detection bounding boxes to obtain the face detection results.

[0216] S109. Send the face detection results to the face optimization module.

[0217] Therefore, in this embodiment of the application, before clustering high-confidence face detection boxes to obtain the group photo line, the low-confidence face detection boxes are matched with the set of subject detection boxes. If the matching result satisfies the above-mentioned spatial positional relationship constraints, the low-confidence face detection boxes are recalled to the high-confidence face detection boxes, thereby improving the accuracy of face detection box prediction, improving the accuracy of subsequent face optimization algorithms, and improving the user's photo-taking experience.

[0218] In one implementation, since the pose of the subject in a group photo scene is adjusted during the shooting process, the confidence of face detection boxes in the current frame image can be combined with the confidence of face detection boxes in historical frame images to predict and recall face detection boxes. This face detection method includes the following steps:

[0219] S101. The application is started in response to the first operation command triggered by the user.

[0220] S102. Acquire the image stream in response to a second operation command triggered by the user.

[0221] S103. Obtain the current frame image.

[0222] S104. Perform face detection box prediction on the current frame image to obtain a set of face detection boxes.

[0223] S105. The face detection box set is divided based on the first confidence threshold and the second confidence threshold to obtain a high-confidence face detection box set and a low-confidence face detection box set.

[0224] S301. Obtain at least one historical frame image.

[0225] Understandably, in photography scenarios, photographers typically take multiple shots of the same scene and select the best image as the final image. Furthermore, because the intervals between shots are short, the range of movement of the same subject's face is relatively small. Therefore, the face detection bounding boxes of the same subject in multiple historical frames can be matched with the face detection bounding boxes in the current frame to determine whether the face detection bounding box in the current frame represents a missed real face.

[0226] Among them, the multi-frame historical image can be generated by the user clicking the shutter button multiple times, or it can be obtained when the user presses and holds the shutter button, or it can be obtained from the preview video stream obtained by the camera application. This application embodiment does not limit this.

[0227] For example, Figure 4 The image shown in the middle (1) interface can be considered as Figure 5 The historical frame image shown in the interface (1).

[0228] S302. Predict face detection boxes on historical frame images to obtain a set of historical face detection boxes.

[0229] Specifically, a face recognition model is used to predict face detection boxes in historical frame images to obtain a set of historical face detection boxes.

[0230] For example, such as Figure 4 All the dashed boxes in the interface shown in (2) are the set of historical face detection boxes that have been identified.

[0231] S303. Match each face detection box in the low-confidence face detection box set with the historical face detection box set. When a match is successful and the historical face detection box is a high-confidence face detection box, recall the corresponding face detection box to the high-confidence face detection box set.

[0232] Specifically, each face detection box in the low-confidence face detection box set is associated with its corresponding spatial coordinates with the spatial coordinates in the historical face detection box set. When a historical face detection box is a high-confidence face detection box, it is recalled to the high-confidence face detection box set.

[0233] For example, the above-mentioned step of associating using spatial coordinates can be measured using overlap data. A higher IOU value between two face detection boxes indicates a higher degree of overlap; when the IOU exceeds a preset threshold, the two face detection boxes can be considered to belong to the same subject.

[0234] For example, the above method for calculating overlap can be found in [reference needed]. Figure 8 The aforementioned overlap threshold can be set to 0.5.

[0235] It is understandable that the aforementioned overlap preset threshold can be set according to different group photo scenarios, and different thresholds will increase / or decrease the amount of computation during recall.

[0236] For example, see Figure 9 In interfaces 50a and 50b, the face detection bounding box of person RW1 is a high-confidence face detection box in the historical interface shown in interface 50a, but a low-confidence face detection box in the current interface shown in interface 50b. Based on the spatial position of the face detection boxes, it is determined that both are for the same subject. Therefore, in the current interface shown in interface 50b, the face detection box of person RW1 is recalled to the set of high-confidence face detection boxes.

[0237] S106. Cluster the face detection boxes in the high-confidence face detection box set to obtain the average width and height of the face detection boxes of all face boxes on the group photo line.

[0238] S107. Based on the group photo line and its corresponding average width and height, recall the face detection boxes in the low-confidence face detection box set to the high-confidence face detection box set.

[0239] S108. Perform nonmaximum suppression on the high-confidence face detection bounding boxes to obtain the face detection results.

[0240] S109. Send the face detection results to the face optimization module.

[0241] Therefore, in this embodiment of the application, before clustering high-confidence face detection boxes to obtain the group photo line, high-confidence face detection boxes in historical frame images are matched with low-confidence face detection boxes in the current frame image, thereby recalling low-confidence face detection boxes, which improves the accuracy of face detection box prediction, improves the accuracy of subsequent face optimization algorithms, and enhances the user's photo-taking experience.

[0242] Furthermore, it is understood that, in order to achieve the aforementioned functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0243] Furthermore, it should be noted that in practical application scenarios, the face detection methods provided in the above embodiments, implemented by electronic devices, can also be executed by a chip system included in the electronic device. This chip system may include a processor. The chip system can be coupled to a memory, enabling it to call computer programs stored in the memory during runtime to implement the steps executed by the electronic device. The processor in this chip system can be an application processor or a non-application processor.

[0244] In addition, this application embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the above-mentioned related method steps to implement the face detection method in the above embodiment.

[0245] In addition, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the above-mentioned related steps to implement the face detection method in the above embodiments.

[0246] In addition, embodiments of this application also provide a chip (which may also be a component or module), the chip may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the processing circuits communicate with each other through internal connection paths, the processing circuits execute the above-mentioned related method steps to implement the face detection method in the above embodiments, so as to control the receiving pins to receive signals and control the transmitting pins to transmit signals.

[0247] Furthermore, as can be seen from the above description, the electronic devices, computer-readable storage media, computer program products, or chips provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0248] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A face detection method, characterized in that, The method includes: Based on a pre-trained face detection model, face detection boxes are predicted for the current frame image to obtain a set of face detection boxes. The information of each face detection box in the set includes confidence, width, height and first spatial coordinates. Based on a first preset confidence threshold, a second preset confidence threshold, and the confidence level of each face detection box, the face detection box set is divided into a high-confidence face detection box set and a low-confidence face detection box set, wherein the first preset confidence threshold is greater than the second preset confidence threshold. Cluster each face detection box in the set of high-confidence face detection boxes to obtain at least one composite line and its corresponding second spatial coordinates; Calculate the first average width and first average height of the face detection box along each of the group photo lines; The first spatial coordinates, the width, and the height of each face detection box in the low-confidence face detection box set are compared with the second spatial coordinates, the first average width value, and the first average height value of the corresponding group photo line. If the comparison result satisfies the spatial positional relationship constraint of the first group photo line, the corresponding face detection box is recalled to the high-confidence face detection box set.

2. The method according to claim 1, characterized in that, The process involves clustering each face detection box in the set of high-confidence face detection boxes to obtain at least one grouping line and its corresponding second spatial coordinates. Calculate the first average width and first average height of the face detection bounding box along each of the group photo lines, including: Clustering algorithms are used to cluster the y-coordinates of the first spatial coordinates of each face detection box in the high-confidence face detection box set, and at least one cluster center line obtained by clustering is set as the group photo line. Calculate the first average width and the first average height of the face detection box corresponding to each of the group photo lines.

3. The method according to claim 2, characterized in that, The step of using a clustering algorithm to cluster the y-coordinates of the first spatial coordinates of each face detection box in the high-confidence face detection box set, and setting at least one cluster center line obtained from the clustering as the group photo line, includes: The k-means clustering algorithm is used to cluster the y-coordinate of each face detection box in the set of high-confidence face detection boxes, and the elbow method is used to segment the clustering results to obtain at least one composite line.

4. The method according to claim 1, characterized in that, The step of recalling the corresponding face detection box to the high-confidence face detection box set when the comparison result satisfies the spatial positional relationship constraint of the first group photo line includes: Obtain the first face detection box currently in the set of low-confidence face detection boxes; If the second spatial coordinate of any of the group photo lines falls within the first spatial coordinate of the current first face detection frame, and it is determined that the current first face detection frame is on the group photo line, and the difference between the width of the current first face detection frame and the first average width value of the group photo line is within the preset difference range corresponding to the group photo line, and the difference between the height of the current first face detection frame and the first average height value of the group photo line is within the preset difference range, then it is determined that the comparison result satisfies the spatial positional relationship constraint condition of the first group photo line. The current first face detection box is recalled to the high-confidence face detection box set until all the first face detection boxes in the low-confidence face detection box set have been traversed.

5. The method according to claim 4, characterized in that, The preset difference range includes: Obtain the largest and smallest face detection boxes in the set of face detection boxes; Remove the largest and smallest face detection boxes from the multiple face detection boxes along each of the group photo lines; Calculate the second average width and second average height of the remaining face detection boxes on each of the aforementioned group photo lines; Calculate the ratio of the second average width and the second average height of each of the group photo lines to the width and height of the largest face detection box and the smallest face detection box, respectively; The maximum width-to-height ratio value among the multiple width-to-height ratio values ​​calculated for each group photo line is set as the preset difference range for its corresponding group photo line.

6. The method according to claim 1, characterized in that, Before clustering each face detection box in the high-confidence face detection box set to obtain at least one composite line and its corresponding second spatial coordinates, the method further includes: Based on the pre-trained subject detection model, the current frame image is used to predict subject detection boxes to obtain a set of subject detection boxes; Match each face detection box in the low-confidence face detection box set with each subject detection box in the subject detection box set; If a match is successful, the corresponding face detection box is recalled to the set of high-confidence face detection boxes.

7. The method according to claim 6, characterized in that, The step of matching each face detection box in the low-confidence face detection box set with each subject detection box in the subject detection box set includes: If the current second face detection box in the low-confidence face detection box set is inside any subject detection box in the subject detection box set, and the current second face detection box is located in the upper half of the subject detection box, and the width of the current second face detection box is less than 1 / 2 of the subject detection box, then a match is determined to be successful.

8. The method according to claim 6, characterized in that, When the face detection model supports the subject detection task, the subject detection model is a submodule of the face detection model.

9. The method according to claim 1, characterized in that, Before clustering each face detection box in the high-confidence face detection box set to obtain at least one composite line and its corresponding second spatial coordinates, the method further includes: At least one historical frame image is acquired, and face detection boxes are predicted on the historical frame image based on the face detection model to obtain a set of historical face detection boxes corresponding to the historical frame image. The information of each face detection box in the set of historical face detection boxes includes confidence and third space coordinates. Based on the third spatial coordinates and the first spatial coordinates, each face detection box in the historical face detection box set and the low-confidence face detection box set is matched; If a match is successful, obtain the corresponding historical face detection box; If the confidence level of the historical face detection box is greater than the first preset confidence threshold, the corresponding face detection box is recalled to the high confidence face detection box set.

10. The method according to claim 9, characterized in that, The matching of each face detection box in the historical face detection box set and the low-confidence face detection box set based on the third spatial coordinates and the first spatial coordinates includes: Based on the overlap between the third spatial coordinates and the first spatial coordinates, the overlap between the current historical face detection box in the historical face detection box set and the current face detection box in the low-confidence face detection box set is calculated. If the overlap is greater than a preset overlap threshold, it is determined that the current historical face detection box and the current face detection box are successfully matched.

11. The method according to claim 1, characterized in that, After recalling the corresponding face detection box to the high-confidence face detection box set when the comparison result satisfies the spatial positional relationship constraint of the first group photo line, the method further includes: Non-maximum suppression is applied to the face detection boxes in the set of high-confidence face detection boxes to obtain the final face detection result.

12. An electronic device, characterized in that, The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, which, when executed by the processor, cause the electronic device to perform the face detection method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The method includes a computer program that, when run on an electronic device, causes the electronic device to perform the face detection method as described in any one of claims 1 to 11.

Images