Image processing method and apparatus therefor, storage medium, program product

Abstract

This invention discloses an image processing method, apparatus, storage medium, and program product. After acquiring an infrared image and a depth image with corresponding pixel relationships, the method first performs face detection on the infrared image to obtain a face detection bounding box for the target face. Then, the face detection bounding box is mapped onto the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, based on the target coordinate information, the face region is extracted from the depth image to obtain a depth map of the target face region. Then, contour fitting is performed on the target face in the depth map of the face region to obtain a face contour curve frame, which is then used for subsequent image processing. This invention can improve the accuracy of face region detection under uneven ambient lighting conditions, thereby improving the adjustment effect of FaceAE processing. This invention can be widely applied to image processing technologies in various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, storage medium, and program product. Background Technology

[0002] Face Auto Exposure (FaceAE) technology can automatically adjust the brightness of a face image. Therefore, if a face image is processed with FaceAE before recognition, the recognition effect and accuracy can be improved.

[0003] Before performing FaceAE processing on a face image, it is often necessary to detect the face region in the face image first. However, when the ambient light is uneven, the face region often appears to be bright on one side and dark on the other. In this case, there is no clear boundary between the face region and the background region, which makes it impossible for traditional face detection methods to accurately detect the face region, thus affecting the adjustment effect of FaceAE processing. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides an image processing method, apparatus, storage medium, and program product that can improve the detection accuracy of face regions under uneven ambient lighting conditions, thereby improving the adjustment effect of FaceAE processing.

[0006] On one hand, embodiments of the present invention provide an image processing method, including the following steps:

[0007] A target face image to be processed is obtained, the image to be processed includes an infrared image and a depth image, and the pixels of the infrared image and the pixels of the depth image have a corresponding relationship;

[0008] Face detection is performed on the infrared image to obtain the face detection bounding box of the target face;

[0009] The face detection box is mapped onto the depth image to obtain the target coordinate information of the face detection box in the depth image;

[0010] Based on the target coordinate information, the face region is extracted from the depth image to obtain the face region depth map of the target face.

[0011] The target face in the depth map of the face region is fitted with a contour to obtain a face contour curve frame, which is then used for subsequent image processing.

[0012] On the other hand, embodiments of the present invention also provide an image processing apparatus, including:

[0013] An image acquisition unit is used to acquire a target face image to be processed, the image to be processed including an infrared image and a depth image, wherein the pixels of the infrared image and the pixels of the depth image have a corresponding relationship;

[0014] A face detection unit is used to perform face detection on the infrared image to obtain a face detection box of the target face;

[0015] A coordinate mapping unit is used to map the face detection box to the depth image to obtain the target coordinate information of the face detection box in the depth image;

[0016] A face extraction unit is used to extract the face region from the depth image based on the target coordinate information to obtain a face region depth map of the target face.

[0017] An image exposure unit is used to perform contour fitting on the target face in the depth map of the face region to obtain a face contour curve frame, so as to use the face contour curve frame for subsequent image processing.

[0018] Optionally, the face extraction unit is further configured to:

[0019] The depth image is segmented into foreground and background regions to obtain a denoised depth map;

[0020] Based on the denoised depth map and the target coordinate information, the face region is extracted from the depth image to obtain the face region depth map of the target face.

[0021] Optionally, the face extraction unit is further configured to:

[0022] The target face region is extracted from the denoised depth map based on the target coordinate information.

[0023] The target face region is copied into the target region of the depth image based on the target coordinate information to obtain the face region depth map of the target face.

[0024] Optionally, the face extraction unit is further configured to:

[0025] Set all depth pixel values ​​in the depth image to the first target value to obtain a backup depth map;

[0026] The target face region is copied to the target region in the backup depth map based on the target coordinate information to obtain the face region depth map of the target face.

[0027] Optionally, the face extraction unit is further configured to:

[0028] Obtain the foreground segmentation threshold and background segmentation threshold;

[0029] In the depth image, depth pixel values ​​that are less than the foreground segmentation threshold are set as the second target value, and depth pixel values ​​that are greater than the background segmentation threshold are set as the second target value to obtain a denoised depth map.

[0030] Optionally, the face extraction unit is further configured to:

[0031] Obtain the average depth pixel value of the depth image;

[0032] The foreground segmentation threshold is calculated based on the mean depth pixel value and the first depth threshold.

[0033] The background segmentation threshold is calculated based on the mean value of the depth pixels and the second depth threshold.

[0034] Optionally, the image exposure unit is further configured to:

[0035] Contour extraction is performed on the target face in the depth map of the face region to obtain face contour coordinate information;

[0036] Based on the facial contour coordinate information, the target face is fitted with a contour to obtain a facial contour curve frame.

[0037] Optionally, the image exposure unit is further configured to:

[0038] The depth map of the face region is binarized to obtain a binarized depth map;

[0039] The binarized depth map is converted to a different data type to obtain the target depth map;

[0040] The contour of the target face in the target depth map is extracted to obtain the face contour coordinate information.

[0041] Optionally, the image to be processed further includes a color image, wherein the pixels of the color image and the pixels of the depth image have a corresponding relationship; the image exposure unit is further used for:

[0042] The face contour curve is mapped onto the color image to obtain the target contour curve.

[0043] Automatic face exposure is performed on the target face in the color image based on the target contour curve to obtain the target image.

[0044] Optionally, the image exposure unit is further configured to:

[0045] Based on the target contour curve frame, the brightness information of the target face in the color image is statistically analyzed to obtain brightness statistical information;

[0046] Based on the brightness statistics, the target face in the color image is automatically exposed to obtain the target image.

[0047] Optionally, the image exposure unit is further configured to:

[0048] The automatic exposure adjustment parameters are calculated based on the brightness statistics.

[0049] The target face in the color image is automatically exposed according to the automatic exposure adjustment parameters to obtain the target image.

[0050] On the other hand, embodiments of the present invention also provide an image processing apparatus, including:

[0051] At least one processor;

[0052] At least one memory for storing at least one program;

[0053] The image processing method as described above is implemented when at least one of the programs is executed by at least one of the processors.

[0054] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the image processing method as described above.

[0055] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of a computer device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the computer device to perform the image processing method as described above.

[0056] The embodiments of the present invention include at least the following beneficial effects: After obtaining an infrared image and a depth image showing the correspondence between pixels of the target face, face detection is first performed on the infrared image to obtain a face detection bounding box of the target face. Then, the face detection bounding box is mapped onto the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Then, contour fitting is performed on the target face in the face region depth map to obtain a face contour curve frame, which is then used for subsequent image processing. Since neither the infrared image nor the depth image is affected by... The intensity of ambient light has an impact, and depth images can reflect the geometric shape information of the target face. Therefore, when the ambient light is uneven and the face area and background area of ​​the face image are not clearly defined, infrared images can obtain more accurate face detection boxes for the target face. Then, based on the face detection box and the depth image, a depth map of the target face area can be effectively obtained. Subsequently, based on the depth map of the face area, a more accurate face contour curve can be obtained, thereby improving the accuracy of face area detection. In addition, the more accurate face contour curve can be used to improve the adjustment effect of FaceAE processing.

[0057] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0058] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0059] Figure 1 This is a schematic diagram of a face image captured in a side-lit scene according to an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of another face image captured in a side-lit scene according to an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of another implementation environment provided by an embodiment of the present invention;

[0063] Figure 5 This is a flowchart of an image processing method provided in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of an infrared image including a rectangular detection box provided by an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of mapping a face detection bounding box from an infrared image to a depth image, provided by an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of a depth image of a target human face provided in an embodiment of the present invention;

[0067] Figure 9 Based on Figure 8 A schematic diagram of the depth map of the face region obtained after face region extraction processing;

[0068] Figure 10 This is a comparison image of a color image and a depth image provided in an embodiment of the present invention;

[0069] Figure 11 This is a complete flowchart of an image processing method provided in an embodiment of the present invention;

[0070] Figure 12 This is a schematic diagram of a rectangle smaller than the actual face area provided in an embodiment of the present invention;

[0071] Figure 13 This is a schematic diagram of a rectangle larger than the actual face area provided in an embodiment of the present invention;

[0072] Figure 14 This is a comparison diagram of the region using a rectangular frame as statistical brightness information and the region using an elliptical curve frame as statistical brightness information, provided in an embodiment of the present invention.

[0073] Figure 15 This is a comparison image of a face region depth map and a color image mapped with a target contour curve provided in an embodiment of the present invention;

[0074] Figure 16 This is a comparison diagram of the effects of a target image obtained by FaceAE processing using a rectangular frame and a target image obtained by FaceAE processing using an elliptical curve frame, provided in an embodiment of the present invention.

[0075] Figure 17 This is a schematic diagram of an image processing device provided in an embodiment of the present invention;

[0076] Figure 18 This is a schematic diagram of another image processing device provided in an embodiment of the present invention. Detailed Implementation

[0077] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present invention, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0078] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0080] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0081] 1) Infrared images are images formed based on the sensing of infrared radiation. Compared with visible light, infrared imaging is not affected by the intensity of ambient light, so clear infrared images can be obtained even in low-light environments. Infrared image sensors sense infrared radiation formed by thermal radiation within their field of view, convert the corresponding infrared signals into electrical signals, and then process them through an image signal processor to form an infrared image of the target object.

[0082] 2) Depth Image, also known as Range Image, is an image that uses the distance (or depth) from the image acquisition device to various points in the scene as pixel values. Depth Image can reflect the geometry of the visible surface of the scene.

[0083] 3) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0084] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0085] 4) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0086] 5) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0087] 6) Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0088] 7) Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy.

[0089] Before performing FaceAE processing on a face image, it is often necessary to detect the face region within the image. Common face detection methods include color-based face contour detection, edge-fitting-based face contour detection, and deep learning-based face contour detection. However, when the ambient lighting is uneven, the face region often appears unevenly lit, with one side bright and the other dark. For example… Figure 1 and Figure 2As shown, when users make facial recognition payments outdoors or in semi-outdoor environments, if the lighting is uneven, the captured facial image will exhibit a "two-tone" characteristic (one side bright, one side dark). In this situation, the bright edges of the face will be overexposed, while the dark edges will be underexposed, resulting in no clear boundary between the face and background areas. Consequently, neither color-based nor edge-fitting facial contour detection schemes can accurately identify the facial region in the image. Furthermore, overexposure leads to the loss of facial details, while underexposure affects facial contrast, thus reducing the accuracy of the facial recognition algorithm and impacting the subsequent FaceAE processing adjustments. Deep learning-based facial contour detection schemes can more accurately identify the facial region in the image; however, these schemes require significant computing power. When real-time processing and result feedback are needed, the computing power requirements increase further, leading to higher hardware costs.

[0090] To improve the accuracy of face region detection under uneven ambient lighting conditions with lower hardware costs, thereby enhancing the adjustment effect of FaceAE processing, this invention provides an image processing method, an image processing device, a computer-readable storage medium, and a computer program product. After acquiring an infrared image and a depth image showing the correspondence between pixels of the target face, face detection processing is first performed on the infrared image to obtain a face detection bounding box. Then, the coordinate information of the face detection bounding box in the infrared image is mapped to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction processing is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Finally, the face region depth map... The target face in the image is fitted with a facial contour to obtain a facial contour bounding box. Since both infrared and depth images are unaffected by ambient light intensity, and the depth image reflects the geometric shape of the target face, infrared images can obtain a more accurate face detection bounding box when the face region and background region are not clearly defined due to uneven ambient light. Based on this bounding box and the depth image, a depth map of the target face region can be effectively obtained. Then, based on this depth map, a more accurate facial contour bounding box can be obtained, thus improving the accuracy of face region detection. This more accurate contour bounding box can then be used to improve the adjustment effect of FaceAE processing. Furthermore, none of the steps in obtaining the facial contour bounding box require high computing power from the corresponding hardware, thus achieving improved accuracy in face region detection with relatively low hardware costs.

[0091] The solutions provided in the embodiments of the present invention relate to technologies such as image processing in artificial intelligence, and are specifically described through the following embodiments.

[0092] Figure 3 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 3 The implementation environment includes the first terminal 101.

[0093] The first terminal 101 may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Optionally, the first terminal 101 may be equipped with an application for face recognition, and the first terminal 101 may use the application to detect the target face in the image to be processed.

[0094] The first terminal 101 has at least the functions of acquiring infrared and depth images of the target face and detecting the facial contour of the target face based on the infrared and depth images. For example, after acquiring the infrared and depth images of the target face, it can detect the facial contour of the target face based on the infrared and depth images to obtain a facial contour curve frame, and then perform FaceAE processing on the target face based on the facial contour curve frame.

[0095] Reference Figure 3 As shown, in one application scenario, it is assumed that the first terminal 101 is a payment device for facial recognition payment, and the first terminal 101 is equipped with a camera for capturing the user's facial image and an application for facial detection of the facial image. In response to a user making a facial recognition payment using the payment device, the first terminal 101 uses a camera to capture an infrared image and a depth image of the user, where the pixels of the infrared image and the depth image have a corresponding relationship. After acquiring the user's infrared and depth images, the first terminal 101 performs face detection processing on the infrared image to obtain the user's face detection bounding box. Then, it maps the coordinate information of the face detection bounding box in the infrared image to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, it performs face region extraction processing on the depth image based on the target coordinate information to obtain the user's face region depth map. In response to obtaining the user's face region depth map, the first terminal 101 performs face contour fitting processing on the target face in the face region depth map to obtain a face contour curve frame. In response to obtaining the face contour curve frame, the first terminal 101 performs FaceAE processing on the user's face image based on the face contour curve frame to obtain a target image. When the target image is obtained, the first terminal 101 performs facial recognition processing on the user based on the target image. When the user is confirmed to be recognized, the first terminal 101 initiates a transaction payment request to realize the user's facial recognition payment operation.

[0096] Figure 4This is a schematic diagram of another implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 4 The implementation environment includes a second terminal 102 and a server 103, which are directly or indirectly connected via wired or wireless communication. The second terminal 102 and the server 103 can be nodes in a blockchain, but this embodiment does not specifically limit this.

[0097] The second terminal 102 may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Optionally, the second terminal 102 may be equipped with an application for capturing facial images. The second terminal 102 can use the application to upload the image to be processed, including the target face, to the server 103, so that the server 103 can perform face detection on the target face in the image to be processed.

[0098] The second terminal 102 has at least the functions of acquiring infrared and depth images of the target face and uploading the infrared and depth images to the server 103. For example, after acquiring the infrared and depth images of the target face, it can upload the infrared and depth images to the server 103, so that the server 103 can detect the facial contour of the target face based on the infrared and depth images to obtain a facial contour curve box. Then, it can receive the facial contour curve box sent by the server 103 and then perform FaceAE processing on the target face based on the facial contour curve box.

[0099] Server 103 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0100] Server 103 has at least the functions of acquiring infrared and depth images of the target face and detecting the facial contour of the target face based on the infrared and depth images. For example, after receiving the infrared and depth images sent by the second terminal 102, it can detect the facial contour of the target face based on the infrared and depth images to obtain a facial contour curve frame, and then send the facial contour curve frame to the second terminal 102 so that the second terminal 102 can perform FaceAE processing on the target face based on the facial contour curve frame.

[0101] Reference Figure 4As shown, in one application scenario, assuming the second terminal 102 is a vehicle-mounted terminal, and the second terminal 102 is equipped with a camera for capturing user facial images and an application for facial recognition, the user can log in to the vehicle system by scanning their face through this application. In response to the user's login operation to the vehicle system, the second terminal 102 uses the camera to capture the user's infrared and depth images, where the pixels of the infrared image and the depth image have a corresponding relationship. After acquiring the user's infrared and depth images, the second terminal 102 uploads them to the server 103. In response to receiving the infrared and depth images, the server 103 first performs face detection processing on the infrared image to obtain the user's face detection bounding box, then maps the coordinate information of the face detection bounding box in the infrared image to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image, and then processes the depth image according to the target coordinate information. The system extracts and processes the user's face region to obtain a depth map of the face region. Then, it performs face contour fitting on the target face in the depth map to obtain a face contour curve. After obtaining the face contour curve, the server 103 sends the face contour curve to the second terminal 102. In response to receiving the face contour curve, the second terminal 102 performs FaceAE processing on the user's face image based on the face contour curve to obtain a target image. Once the target image is obtained, the second terminal 102 performs face recognition processing on the user based on the target image. When the user is recognized, the second terminal 102 confirms that the user has successfully logged in, thus realizing the user's login operation of the vehicle system.

[0102] It should be noted that in various specific embodiments of the present invention, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of the present invention need to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the target object will the necessary target object-related data for the normal operation of the embodiments of the present invention be obtained.

[0103] The embodiments of the present invention can be applied to various scenarios that require facial recognition, including but not limited to facial recognition scenarios in fields such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0104] Figure 5 This is a flowchart of an image processing method provided in an embodiment of the present invention. This image processing method can be executed by a terminal, or by a terminal and a server in cooperation. (Refer to...) Figure 5The image processing method includes, but is not limited to, steps 110 to 150.

[0105] Step 110: Obtain the image to be processed of the target face. The image to be processed includes an infrared image and a depth image. There is a corresponding relationship between the pixels of the infrared image and the pixels of the depth image.

[0106] It should be noted that the method of acquiring the image to be processed will differ when the executing entity of step 110 is different. For example, when the executing entity of step 110 is a terminal, the terminal can acquire the image to be processed by taking a picture with a camera; when the executing entity of step 110 is a server, the server can acquire the image to be processed by transmitting it from the terminal.

[0107] In some possible implementations, the terminal may integrate an RGBD camera, which can simultaneously output color images, infrared images, and depth images. Therefore, the terminal can obtain infrared images and depth images by capturing images with the RGBD camera. Furthermore, there is a correspondence between the pixels of the infrared image and the pixels of the depth image in the obtained infrared and depth images, so that subsequent steps can perform target face detection processing based on the infrared and depth images.

[0108] It should be noted that there is a correspondence between the pixels of the infrared image and the pixels of the depth image. The pixels of the infrared image and the depth image can be completely aligned, or they can not be completely aligned but have a certain mapping relationship. For example, the conversion of pixel positions between the infrared image and the depth image can be achieved through pixel position mapping. No specific limitation is made here.

[0109] In some possible implementations, through pre-calibration processing by the camera module manufacturer, the RGBD camera can synchronize the color image, infrared image, and depth image when acquiring the image to be processed of the target face. In other words, the color image, infrared image, and depth image acquired by the RGBD camera will be synchronized in time and perfectly aligned in pixel position.

[0110] Step 120: Perform face detection on the infrared image to obtain the face detection bounding box of the target face.

[0111] In this step, since an infrared image was acquired in step 110, and the infrared image is formed based on thermal radiation infrared light sensing and is not affected by the intensity of ambient light, face detection processing can be performed on the infrared image to obtain a relatively accurate face detection box of the target face. This allows subsequent steps to obtain a more accurate face contour curve of the target face based on the face detection box, providing an accurate data foundation for subsequent FaceAE processing.

[0112] It should be noted that there are different ways to perform face detection processing on infrared images to obtain the face detection box of the target face. For example, the infrared image can be input into a face detection neural network model to perform face detection processing to obtain the face detection box of the target face; or, the coordinate values ​​of the thermal radiation points of the target face in the infrared image can be obtained first, and then the face detection box of the target face can be determined based on these coordinate values.

[0113] In some possible implementations, the face detection bounding box of the target face obtained by performing face detection processing on the infrared image can be a rectangular detection bounding box that fits the contour of the target face (e.g., Figure 6 (As shown), it can also be an elliptical detection box that matches the contour of the target face, or a detection box whose shape matches the contour of the target face. The appropriate choice can be made according to the actual application situation, and no specific limitation is made here.

[0114] Step 130: Map the face detection box to the depth image to obtain the target coordinate information of the face detection box in the depth image.

[0115] In this step, since the face detection box of the target face in the infrared image was obtained in step 120 and the depth image was obtained in step 110, and there is a correspondence between the pixels of the infrared image and the pixels of the depth image, the coordinate information of the face detection box in the infrared image can be mapped to the depth image to obtain the target coordinate information of the face detection box in the depth image. This allows subsequent steps to perform target face detection processing based on the face detection box of the target face in the depth image.

[0116] In some possible implementations, since infrared images are formed based on thermal radiation infrared light, the edges of the target face in the infrared image will appear diffused, resulting in blurred edges. To avoid losing the edge features of the target face, the face detection box obtained in step 120 may be slightly too large. To obtain more accurate target coordinate information of the face detection box in the depth image, the face detection box can be reduced in size to better match the target face, and then the reduced face detection box is mapped onto the depth image. For example, assuming the edge of the target face in the infrared image diffuses outward by n pixels, to avoid losing the edge features of the target face, the face detection box obtained in step 120 will expand outward by n pixels. To obtain more accurate target coordinate information of the face detection box in the depth image, in step 130, the face detection box is first reduced in size by n pixels, and then the reduced face detection box is mapped onto the depth image to obtain the target coordinate information of the face detection box in the depth image.

[0117] It should be noted that when the face detection bounding box has different shapes, its coordinate information in the infrared image can take different forms. For example, when the face detection bounding box is a rectangular bounding box that fits the contour of the target face, the coordinate information can include the coordinate values ​​of the four corners of the rectangular bounding box, or the coordinate values ​​of any pair of opposite corners. When the face detection bounding box is an elliptical bounding box that fits the contour of the target face, the coordinate information can include the coordinate values ​​of the center point, the major axis, the minor axis, and the rotation angle. When the face detection bounding box has a shape consistent with the contour of the target face, the coordinate information can include the coordinate values ​​of the key points of the target face contour. Similarly, since the target coordinate information of the face detection bounding box in the depth image is obtained by mapping the coordinate information of the face detection bounding box in the infrared image to the depth image, the content included in the target coordinate information of the face detection bounding box in the depth image is the same as that included in the coordinate information of the face detection bounding box in the infrared image. To avoid redundancy, it will not be repeated here.

[0118] It should be noted that when there are different correspondences between the pixels of the infrared image and the pixels of the depth image, there are different ways to map the coordinate information of the face detection box in the infrared image to the depth image. For example, when the pixels of the infrared image and the depth image are perfectly aligned, the coordinate information of the face detection box in the infrared image is consistent with the coordinate information of the face detection box in the depth image. Therefore, the coordinate information of the face detection box in the infrared image can be directly applied to the depth image. Alternatively, when there is a certain mapping relationship between the pixels of the infrared image and the pixels of the depth image, the transformation formula of the pixel coordinates between the infrared image and the depth image can be determined first. Then, the coordinate information of the face detection box in the infrared image can be mapped to the depth image according to the transformation formula to obtain the target coordinate information of the face detection box in the depth image. The transformation formula can be a linear transformation formula or a non-linear transformation formula, which is not specifically limited here. For example, assuming the pixel coordinate transformation formula in the horizontal direction is a linear transformation formula, this transformation formula can be Xd = Xir + ΔPix, where Xd is the target coordinate information of the face detection box in the horizontal direction of the depth image, Xir is the coordinate information of the face detection box in the horizontal direction of the infrared image, and ΔPix is ​​the pixel coordinate offset value, which can be positive or negative. For example Figure 7 As shown, Figure 7 This is a schematic diagram illustrating how the coordinates of a face detection bounding box in an infrared image are mapped to a depth image. Figure 7 In the diagram, the left image shows a schematic of the face detection bounding box in the infrared image. After mapping the coordinate information of the face detection bounding box in the infrared image to the depth image, we can obtain the schematic of the face detection bounding box in the depth image as shown on the right.

[0119] Step 140: Extract the face region from the depth image based on the target coordinate information to obtain the face region depth map of the target face.

[0120] In this step, since the target coordinate information of the face detection box in the depth image is obtained in step 130, and the depth image is an image that uses the distance (or depth) from the image acquisition device to each point in the scene as the pixel value, it is not affected by the intensity of ambient light. Therefore, the face region can be extracted from the depth image according to the target coordinate information to obtain the face region depth map of the target face. This allows subsequent steps to obtain a more accurate face contour curve of the target face based on the face region depth map, providing an accurate data foundation for subsequent FaceAE processing.

[0121] It should be noted that there are different implementation methods for face region extraction processing based on target coordinate information in depth images. For example, the target coordinate information and the depth image can be input into a face extraction neural network model to perform face region extraction processing to obtain a face region depth map of the target face. Alternatively, the target region including the face region can be determined in the depth image based on the target coordinate information. Since there is generally a large difference in depth pixel values ​​between the face region and the background region in the depth image, the range of the face region can be determined based on the depth pixel values ​​in the target region. Then, face region extraction processing can be performed based on this face region range to obtain a face region depth map of the target face. For example... Figure 8 and Figure 9 As shown, Figure 8 This is a schematic diagram of the depth image of a target face in an example. Figure 9 Based on Figure 8 This is a schematic diagram of the depth map of the target face region obtained after face region extraction processing. After obtaining the target coordinate information of the face detection box in the depth image in step 130, the target coordinate information is used to... Figure 8 The depth image shown can be processed to extract the face region, resulting in the following: Figure 9 The image shows the depth map of the target face region.

[0122] Step 150: Perform contour fitting on the target face in the depth map of the face region to obtain the face contour curve frame, which can be used for subsequent image processing.

[0123] In this step, since the depth image contains more and more detailed facial information, after obtaining the depth map of the target face region in step 140, the target face in the depth map of the face region can be subjected to face contour fitting processing to obtain a more accurate face contour curve box of the target face, thereby improving the detection accuracy of the face region. Then, image exposure processing can be performed based on the face contour curve box, thereby improving the adjustment effect of FaceAE processing.

[0124] It should be noted that using the face contour curve for subsequent image processing refers to using the face contour curve after obtaining it. For example, the face contour curve can be used to determine the target area, and then different image processing can be performed on the target face in the target area, such as exposure adjustment, beautification, screenshotting, or blurring.

[0125] It should be noted that there are different ways to perform face contour fitting on a target face in the face region depth map to obtain a face contour curve. For example, one can first extract the key points of the target face contour in the face region depth map, and then use a curve fitting formula (such as the cv2.fitEllipse function for fitting elliptical curves) to perform face contour fitting on these key points to obtain a face contour curve. This face contour curve can be represented as (x... 2 / a 2 )+(y 2 / b 2) = 1, (x, y) are the coordinates of a point on the face contour curve, a is the length of the major semi-axis of the face contour curve, and b is the length of the minor semi-axis of the face contour curve; Alternatively, one can first extract the face contour key points in the face region depth map, then connect these face contour key points to form an initial curve representing the face contour, and then perform detail adjustments on the neighboring regions of this initial curve to obtain the face contour curve. For example, one can first use regression tree knowledge (Ensembles of Regression) The Trees (ERT) algorithm detects several key feature points near the contour of the target face, such as those around the eyes, mouth, and chin. Then, the Catmull-Rom algorithm uses these key feature points to construct an initial curve representing the face contour region. Next, the entire face contour region is divided into many smaller, overlapping square regions along this initial curve, with the center point of each square region lying on the initial curve. Each square region contains a local contour curve. Then, the local contour curves in each square region are smoothed, connecting all the local contour curves in all square regions to form a smooth face contour bounding box. It should be noted that the Catmull-Rom algorithm is used to draw curves based on discrete points.

[0126] It should be noted that different face contour fitting processes can yield different face contour bounding boxes. For example, the obtained face contour bounding box can be an elliptical bounding box that fits the contour of the target face, or it can be a bounding box with a shape consistent with the contour of the target face. The appropriate choice can be made according to the actual application, and no specific limitation is made here. For example, assuming that the cv2.fitEllipse function mentioned above is used to fit the contour of the target face, an elliptical bounding box that fits the contour of the target face can be obtained; on the other hand, assuming that the above-mentioned method of first extracting the key points of the face contour, then connecting them into an initial curve, and then adjusting the initial curve to obtain the face contour bounding box is used, a bounding box with a shape consistent with the contour of the target face can be obtained. It is worth noting that regardless of whether the face outline is an elliptical outline that fits the outline of the target face or a outline whose shape matches the outline of the target face, it can be used to remove background interference information during FaceAE processing. This allows automatic exposure processing to be performed only on the target face within the face outline area during FaceAE processing, thereby improving the adjustment effect of FaceAE processing.

[0127] In this embodiment, using the image processing method including steps 110 to 150 above, after obtaining an infrared image and a depth image showing a correspondence between the pixels of the target face, face detection is first performed on the infrared image to obtain a face detection bounding box for the target face. Then, the face detection bounding box is mapped onto the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Then, contour fitting is performed on the target face in the face region depth map to obtain a face contour curve frame, which is then used for subsequent image processing. Since the infrared image and... Depth images are unaffected by ambient light intensity and can reflect the geometric shape information of the target face. Therefore, when uneven ambient light results in no clear boundary between the face region and the background region in the face image, infrared images can obtain more accurate face detection bounding boxes. Based on these bounding boxes and the depth image, a depth map of the target face region can be effectively obtained. Then, based on this depth map, a more accurate face contour curve can be obtained, thereby improving the accuracy of face region detection. This more accurate face contour curve can then be used to improve the adjustment effect of FaceAE processing. Furthermore, steps 110 to 150 do not require higher computing power from the corresponding hardware, thus achieving improved face region detection accuracy with lower hardware costs.

[0128] In some possible implementations, when performing face region extraction processing on the depth image based on the target coordinate information to obtain the face region depth map of the target face, the depth image can first be segmented into foreground and background regions to obtain a denoised depth map, and then the face region extraction processing can be performed on the depth image based on the denoised depth map and the target coordinate information to obtain the face region depth map of the target face. Since the depth image of the target face contains distance information from the target face to the image sensor and distance information from the background region to the image sensor, and the distance information from the target face to the image sensor includes distance information from the foreground region to the image sensor and geometric shape information of the target face, the distance information from the background region to the image sensor and the distance information from the foreground region to the image sensor can interfere with the face region extraction process. Therefore, in order to obtain a more accurate face region depth map of the target face, we can first perform foreground and background region segmentation processing on the depth image to remove useless noise from the depth image and obtain a denoised depth map. Then, based on the denoised depth map and the target coordinate information, we can perform face region extraction processing on the depth image to obtain a more accurate face region depth map. This allows subsequent steps to obtain a more accurate face contour curve of the target face based on the face region depth map, providing an accurate data foundation for subsequent FaceAE processing.

[0129] In some possible implementations, when performing foreground and background region segmentation on a depth image to obtain a denoised depth map, a foreground segmentation threshold and a background segmentation threshold can be obtained first. Then, in the depth image, depth pixel values ​​smaller than the foreground segmentation threshold are set as a second target value, and depth pixel values ​​larger than the background segmentation threshold are set as the second target value, thereby obtaining a denoised depth map. Since the distance information from the background region to the image acquisition device and the distance information from the foreground region to the image acquisition device contained in the depth image can interfere with the face region extraction process, it is necessary to denoise the depth image first. In order to accurately segment the foreground and background regions of the depth image, the foreground segmentation threshold and the background segmentation threshold can be obtained first. Since the user's head has a certain thickness value, in the process of obtaining the foreground segmentation threshold and the background segmentation threshold, the average depth pixel value of the depth image can be obtained first, and the first depth threshold and the second depth threshold can be determined based on the prior knowledge of the head thickness value (for example, the sum of the first depth threshold and the second depth threshold is equal to the head thickness value in the prior knowledge). Then, the foreground segmentation threshold is calculated based on the average depth pixel value and the first depth threshold, and the background segmentation threshold is calculated based on the average depth pixel value and the second depth threshold. After obtaining the foreground and background segmentation thresholds, in the depth image, regions with depth pixel values ​​less than the foreground segmentation threshold are defined as foreground regions, and regions with depth pixel values ​​greater than the background segmentation threshold are defined as background regions. Then, setting the depth pixel values ​​less than the foreground segmentation threshold as the second target value, and setting the depth pixel values ​​greater than the background segmentation threshold as the second target value, yields a denoised depth map with unwanted noise removed. In some examples, when calculating the foreground segmentation threshold based on the average depth pixel value and the first depth threshold, subtracting the first depth threshold from the average depth pixel value gives the foreground segmentation threshold; similarly, when calculating the background segmentation threshold based on the average depth pixel value and the second depth threshold, adding the second depth threshold to the average depth pixel value gives the background segmentation threshold. The first and second depth thresholds can be the same or different, and can be appropriately selected based on the actual application; no specific limitation is made here. Furthermore, the second target value can also be appropriately selected based on the actual application; for example, it can be set to 0, as long as the depth pixel values ​​of the target face region are clearly distinguishable from the second target value; no specific limitation is made here.

[0130] In some possible implementations, when performing face region extraction processing on the depth image based on the denoised depth map and target coordinate information to obtain the face region depth map of the target face, the target face region can first be cropped from the denoised depth map based on the target coordinate information. Then, the target face region can be copied to the target region in the depth image based on the target coordinate information to obtain the face region depth map of the target face. Since the previous processing yielded a denoised depth map after removing useless noise, which retains accurate geometric shape information of the target face, a more accurate target face region can be cropped from the denoised depth map based on the target coordinate information. In this case, copying the target face region to the target region in the depth image can effectively obtain the face region depth map of the target face, which is beneficial for subsequent steps to obtain a more accurate face contour curve of the target face based on this face region depth map, providing an accurate data foundation for subsequent FaceAE processing.

[0131] It should be noted that although depth images can provide a relatively accurate depth map of the face region, this is achieved in conjunction with infrared images. If only depth images are used to obtain the face region depth map, since the pixel values ​​in the depth image represent the distance between the scene and the camera, if other objects exist at the same distance as the target face, the pixel values ​​of the target face's edges will be identical to those of the other objects. In this case, the depth image will show the target face blending seamlessly with the other objects, making it difficult to identify the face region depth map. In this embodiment, however, the face detection bounding box of the target face is first obtained from the infrared image. This bounding box is then mapped onto the depth image to obtain the target coordinates of the bounding box within the depth image. Finally, the face region depth map is obtained from the depth image based on these target coordinates. Since infrared images are formed based on the sensing of infrared radiation from thermal radiation, even if there are other objects at the same distance from the target face to the camera, the infrared radiation emitted by those objects will be different from that emitted by the face. Therefore, a more accurate face detection bounding box can be obtained through infrared images. Since the target coordinate information of the face detection bounding box in the depth image is obtained by mapping the coordinate information of the face detection bounding box in the infrared image, the influence of other objects can be eliminated in the depth image, thereby obtaining a more accurate face region depth map. Therefore, compared with using only the depth image to obtain the face region depth map, this embodiment can obtain a more accurate face region depth map by combining infrared images and depth images.

[0132] In the process of copying the target face region to the target region in the depth image based on the target coordinate information to obtain the face region depth map of the target face, in order to avoid interference from the foreground and background regions in the depth image, all depth pixel values ​​in the depth image can be set to the first target value (e.g., 0 or other values ​​that can be clearly distinguished from the depth pixel values ​​of the target face region) to obtain the backup depth map. At this time, all depth pixel values ​​in the depth image are the same and are clearly distinguished from the depth pixel values ​​of the target face region obtained earlier. Therefore, the interference from the foreground and background regions can be removed. In this case, the target face region is copied to the target region in the backup depth map based on the target coordinate information to obtain the face region depth map of the target face after removing useless noise.

[0133] The following example illustrates in detail the process of obtaining a depth map of the facial region of a target face.

[0134] Assuming that the first and second depth thresholds are both determined to be 130mm based on prior knowledge of head thickness, then in the process of extracting the face region from the depth image based on the target coordinate information to obtain the face region depth map of the target face, the mean function used to calculate the mean can be used to obtain the mean depth pixel value of the entire depth image. Then, the first depth threshold is subtracted from the mean depth pixel value to obtain the foreground segmentation threshold, and the second depth threshold is added to the mean depth pixel value to obtain the background segmentation threshold. Next, the foreground and background regions are determined based on the foreground and background segmentation thresholds to achieve foreground and background region segmentation. At this point, all depth pixel values ​​corresponding to the foreground region and all depth pixel values ​​corresponding to the background region are set to 0 to obtain a denoised depth map. Then, the target face region is extracted from the denoised depth map based on the target coordinate information. After obtaining the target face region and its corresponding depth pixel information, all depth pixel values ​​in the original depth image are set to 0 to obtain a backup depth map. Then, the target face region is copied to the target region in the backup depth map based on the target coordinate information to obtain the face region depth map of the target face after removing useless noise.

[0135] In some possible implementations, when performing face contour fitting on a target face in a face region depth map to obtain a face contour curve, the face contour can first be extracted from the target face in the face region depth map to obtain face contour coordinate information. Then, face contour fitting is performed on the target face based on the face contour coordinate information to obtain a face contour curve. Since the face region depth map has been cleaned of useless noise in the previous processing, meaning that the face region depth map contains more accurate face region information, face contour extraction on the target face in the face region depth map can obtain more accurate face contour coordinate information. At this time, face contour fitting on the target face based on the face contour coordinate information can obtain a more accurate face contour curve. For example, the cv2.fitEllipse function can be used to perform face contour fitting on the target face to obtain a more accurate face contour curve, thereby improving the accuracy of face region detection. Furthermore, the more accurate face contour curve can be used to improve the adjustment effect of FaceAE processing. To improve the efficiency and accuracy of facial contour extraction, when extracting facial contours from the depth map of the face region, the depth map can be binarized first to obtain a binarized depth map. For example, the `cv2.threshold` function can be used to binarize the depth map. Furthermore, since the acquired depth image is typically a 16-bit unsigned integer, to effectively perform subsequent facial contour extraction and fitting, the binarized depth map can be converted to an 8-bit unsigned integer. Then, facial contour extraction can be performed on the target face in this depth map. For example, the `cv2.findContours` function can be used to extract the contour information of the target face in the depth map, resulting in more accurate facial contour coordinates. It should be noted that the facial contour bounding box obtained by fitting the target face contour can be an elliptical bounding box that fits the contour of the target face, or a bounding box with a shape consistent with the contour of the target face. This facial contour bounding box can be used to remove background interference information during FaceAE processing, so that automatic exposure processing is performed only on the target face within the bounding box, thereby improving the adjustment effect of FaceAE processing. It should also be noted that the `cv2.fitEllipse` function for facial contour fitting, the `cv2.threshold` function for binarization, and the `cv2.findContours` function for extracting facial contour information are all functions supported by OpenCV.OpenCV is a commonly used computer vision library that can be used to implement general image processing and image computation.

[0136] In some possible implementations, the image to be processed for the target face may also include a color image, and the pixels of the color image correspond to the pixels of the depth image. In this case, after performing face contour fitting on the target face in the depth image of the face region to obtain a face contour curve, since this face contour curve is a more accurate face contour curve obtained based on the processing of the infrared image and the depth image, even if the ambient light is uneven and the face region and background region in the color image have no clear boundary, the accurate target face contour curve can be obtained by mapping this face contour curve to the color image. Figure 10 As shown, Figure 10 It is a comparison image of a color image and a depth image, in which, Figure 10 The image on the left is a schematic diagram of a color image. Figure 10 The image on the right is a schematic diagram of a depth image. In a color image, the face area appears unevenly lit, resulting in no clear boundary between the face and background areas. However, in a depth image, a more accurate target face contour curve can be obtained. At this point, automatic face exposure processing can be performed on the target face in the color image based on this contour curve to obtain the target image, thus completing FaceAE processing of the target face. Specifically, when performing automatic face exposure processing on the target face in the color image based on the contour curve to obtain the target image, the brightness information of the target face in the color image can first be statistically analyzed based on the contour curve to obtain brightness statistics. Then, automatic face exposure processing is performed on the target face in the color image based on the brightness statistics to obtain the target image. It should be noted that statistical analysis of brightness information for a target face in a color image based on the target contour curve refers to statistically analyzing the brightness of the target face within the target contour curve area of ​​the color image. This includes analyzing the brightness of each pixel position. This allows for the generation of brightness statistics, which are then used in subsequent steps to perform automatic face exposure processing on the target face in the color image. It should also be noted that there is a correspondence between the pixels in the color image and the pixels in the depth image. This correspondence can be either perfectly aligned or not perfectly aligned, but with a certain mapping relationship. For example, pixel position mapping can be used to convert pixel positions between the color image and the depth image; no specific limitation is made here.

[0137] In some possible implementations, the image to be processed for the target face may also include a black and white image, and the pixels of this black and white image correspond to the pixels of the depth image. It should be noted that when the ambient light is uneven, the black and white image is less affected. Therefore, in this case, after obtaining a face contour curve by fitting the target face in the depth image of the face region, this face contour curve can be mapped onto the black and white image to obtain an accurate target face contour curve. Then, this target contour curve curve is used to perform FaceAE processing on the target face in the black and white image. It should be noted that when the ambient light is uneven, the impact on the black and white image is different from that on the color image. Therefore, during FaceAE processing, the exposure parameters can be adjusted according to the corresponding black and white and color images to improve the image quality of both. For example, in some possible implementations, when performing FaceAE processing on a black and white image, the degree of exposure parameter adjustment can be reduced slightly; when performing FaceAE processing on a color image, the degree of exposure parameter adjustment can be increased slightly.

[0138] In some possible implementations, when performing automatic face exposure processing on a target face in a color image based on brightness statistics to obtain a target image, automatic exposure adjustment parameters can be calculated first based on the brightness statistics. For example, the brightness statistics can be input into an automatic exposure adjustment function to obtain the automatic exposure adjustment parameters. Then, automatic face exposure processing is performed on the target face in the color image based on the automatic exposure adjustment parameters to obtain the target image. The calculation of the automatic exposure adjustment parameters based on brightness statistics can have different implementations, which can be appropriately selected according to the actual application and are not specifically limited here. For example, the automatic exposure adjustment parameters for each pixel within the target contour curve frame can be calculated based on the brightness statistics. Alternatively, the pixels within the target contour curve frame can be divided into blocks based on the brightness statistics, and then the automatic exposure adjustment parameters for each divided pixel block can be calculated based on the brightness statistics. After calculating the automatic exposure adjustment parameters for each pixel within the target contour curve frame based on the brightness statistics, the brightness of each pixel within the target contour curve frame can be adjusted according to the automatic exposure adjustment parameters, reducing the brightness of pixels with higher original brightness values ​​and increasing the brightness of pixels with lower original brightness values, resulting in a target image with improved display effects. After calculating the automatic exposure adjustment parameters for each pixel block based on the brightness statistics, an exposure weight value can be set for each pixel block according to the automatic exposure adjustment parameters. Then, the brightness of each pixel block is adjusted according to the exposure weight value, reducing the overall brightness of pixel blocks with lower exposure weight values ​​and increasing the overall brightness of pixel blocks with higher exposure weight values, thereby obtaining a target image with improved display effect.

[0139] The principle of the image processing method provided in this embodiment of the invention will be fully explained below with a specific example.

[0140] Reference Figure 11 , Figure 11 The complete flowchart of the image processing method provided in the embodiment of the present invention specifically includes the following steps 1001 to 1011.

[0141] Step 1001: Start processing.

[0142] In this step, after the terminal captures the user's face image, or after the server receives the user's face image sent by the terminal, the various steps of the image processing method provided in this embodiment will be executed.

[0143] Step 1002: Obtain the image to be processed of the target face. The image to be processed includes an infrared image, a depth image, and a color image. The pixels of the infrared image, the depth image, and the color image have a corresponding relationship.

[0144] In this step, the pixels of the infrared image, the depth image, and the color image have a corresponding relationship. The pixels of the infrared image, the depth image, and the color image can be completely aligned, or they can not be completely aligned but have a certain mapping relationship. For example, the pixel position conversion between the infrared image, the depth image, and the color image can be achieved through pixel position mapping.

[0145] Step 1003: Perform face detection processing on the infrared image to obtain the face detection bounding box of the target face.

[0146] In this step, since infrared images are formed based on thermal radiation infrared light sensing and are not affected by the intensity of ambient light, a relatively accurate face detection bounding box for the target face can be obtained by performing face detection processing on the infrared image.

[0147] In some possible implementations, when performing face detection processing on infrared images, a face detector can be initialized first, then a pre-trained face detection neural network model can be loaded into the face detector, and then the infrared image can be input into the face detection neural network model for face detection processing to obtain the face detection box of the target face.

[0148] In this step, the face detection box obtained by performing face detection processing on the infrared image can be a rectangular detection box that matches the contour of the target face.

[0149] It should be noted that currently used face detection algorithms generally output rectangular bounding boxes to label faces. However, due to the influence of labeled data and actual scene pose, the bounding boxes output by commonly used face detection algorithms often deviate from the actual face regions. For example... Figure 12 and Figure 13 As shown, Figure 12 This is a diagram showing a rectangle smaller than the actual face area. Figure 13This is a diagram illustrating a rectangle larger than the actual face area. Using a rectangle that is too small to collect brightness information about the face area can lead to some facial information being missed in the statistics. This can cause a deviation between the target adjustment direction and the actual brightness of the face in subsequent FaceAE processing, affecting the adjustment effect of FaceAE. Conversely, using a rectangle that is too large can result in more background interference being included in the statistics, similarly causing a deviation between the target adjustment direction and the actual brightness of the face in subsequent FaceAE processing, affecting the adjustment effect of FaceAE. However, in this step, because the obtained face detection bounding box is a rectangle that fits the contour of the target face, the problem of an overly large or undersized face detection bounding box affecting the adjustment effect of FaceAE processing is avoided.

[0150] Step 1004: Perform foreground and background region segmentation on the depth image to obtain a denoised depth map.

[0151] It should be noted that step 1004 can be executed before step 1003, after step 1003, or simultaneously with step 1003. The appropriate choice can be made according to the actual application situation, and no specific limitation is made here.

[0152] In some possible implementations, the mean depth pixel value of the depth image can be obtained first, and a first depth threshold and a second depth threshold can be determined based on prior knowledge of the head thickness value. Then, the mean depth pixel value is subtracted from the first depth threshold to obtain the foreground segmentation threshold, and the mean depth pixel value is added to the second depth threshold to obtain the background segmentation threshold. Next, in the depth image, all depth pixel values ​​less than the foreground segmentation threshold and depth pixel values ​​greater than the background segmentation threshold are set to 0 to obtain a denoised depth map with useless noise removed.

[0153] Step 1005: Map the coordinate information of the face detection box in the infrared image to the depth image to obtain the target coordinate information, and perform face region extraction processing on the denoised depth map based on the target coordinate information to obtain the face region depth map of the target face.

[0154] In this step, the pixels of the infrared image, the depth image, and the color image are perfectly aligned, and their temporal sequences are synchronized. Therefore, the coordinate information of the face detection box in the infrared image can be mapped to the depth image to obtain the target coordinate information. Since the depth image uses the distance (or depth) from the image acquisition device to each point in the scene as the pixel value, it is not affected by the intensity of ambient light. Therefore, the face region can be extracted from the denoised depth map based on the target coordinate information to obtain the face region depth map of the target face. This allows subsequent steps to obtain a more accurate face contour curve of the target face based on the face region depth map, providing an accurate data foundation for subsequent FaceAE processing.

[0155] Step 1006: Extract the face contour of the target face in the depth map of the face region to obtain the face contour coordinate information.

[0156] In this step, since a face region depth map with useless noise removed was obtained in the previous steps, the target face in the face region depth map can be processed to extract the face contour, resulting in more accurate face contour coordinate information.

[0157] In some possible implementations, the depth map of the face region can be binarized first to obtain a binarized depth map, and then the data type of the binarized depth map can be converted to obtain a target depth map. Next, the target face in the target depth map can be extracted to obtain more accurate face contour coordinate information.

[0158] Step 1007: Perform face contour fitting on the target face based on the face contour coordinate information to obtain the face contour curve frame.

[0159] In this step, since more accurate facial contour coordinate information was obtained in the previous steps, facial contour fitting processing can be performed on the target face based on the facial contour coordinate information to obtain a more accurate facial contour curve frame. This more accurate facial contour curve frame can then be used in subsequent steps to improve the adjustment effect of FaceAE processing.

[0160] In this step, the face contour curve obtained by performing face contour fitting on the target face is an elliptical curve that fits the contour of the target face.

[0161] It's important to note that currently used face detection algorithms output rectangular bounding boxes to label faces. Therefore, existing FaceAE algorithms typically use these rectangular bounding boxes as the region for calculating brightness information. However, using rectangular bounding boxes for brightness calculation includes additional background interference (such as background interference in the four corners of the rectangle), which affects the brightness calculation of the face region by the FaceAE algorithm, thus impacting the adjustment effect of FaceAE processing. In this step, because the obtained face contour bounding box is an elliptical bounding box that fits the contour of the target face, it can eliminate the interference caused by the background environment. This limits the region for calculating brightness information by the FaceAE algorithm to the elliptical region of the face, thereby improving the adjustment effect of FaceAE processing, improving the recognition accuracy of the face recognition algorithm, and ultimately improving the user experience. It should be noted that when the FaceAE algorithm statistically analyzes the brightness information within the elliptical region of a face, it first determines the range of this region. For example, it first determines the coordinates of the center point, the length of the major axis, the length of the minor axis, and the deflection angle of the elliptical curve corresponding to the face contour. Then, based on these coordinates, the specific position of the elliptical curve in the face image is determined, thus obtaining the range of the brightness information statistical region. Next, based on this range, the brightness information in the face image is collected, and the exposure of the face image is adjusted according to the collected brightness information. For example... Figure 14 As shown, Figure 14 This is a comparison chart showing the regions using rectangular frames as statistical brightness information and regions using elliptical curve frames as statistical brightness information. Figure 14 The left-hand image is a schematic diagram using rectangular boxes as the area for statistical brightness information. Figure 14 The right-hand image is a schematic diagram using an elliptical curve frame as the area for statistical brightness information. Figure 14 As shown in the two diagrams, when a rectangular frame is used as the area for statistical brightness information, the four corners of the rectangular frame contain background interference information; while when an elliptical curve frame is used as the area for statistical brightness information, the background interference information is eliminated. Therefore, using an elliptical curve frame as the area for statistical brightness information is beneficial to improving the adjustment effect of FaceAE processing.

[0162] Step 1008: Map the face contour curve to the color image to obtain the target contour curve, and perform brightness information statistics and analysis on the target face in the color image based on the target contour curve to obtain the automatic exposure adjustment parameters.

[0163] In this step, because the pixels of the depth image and the color image are perfectly aligned, and the face contour outline is obtained more accurately based on the processing of the infrared and depth images, even if the ambient light is uneven and the face area and background area in the color image are not clearly defined, the accurate target face contour outline can be obtained by mapping the face contour outline to the color image. Then, based on the target contour outline, the brightness information of the target face in the color image is statistically analyzed to obtain the automatic exposure adjustment parameters for the face area within the target contour outline range. This allows subsequent steps to perform targeted FaceAE processing on the target face based on these automatic exposure adjustment parameters. For example... Figure 15 As shown, Figure 15 It is a comparison image of the depth map of the face region and a color image mapped with the contour curve of the target. Figure 15 The image on the left is a schematic diagram of the depth map of the face region. Figure 15 The image on the right is a schematic diagram of a color image mapping the contour curve of the target. According to... Figure 15 As can be seen from the face region depth map, the face region depth map contains the facial contour of the target face. Therefore, in the analysis of... Figure 15 After extracting the facial contour from the depth map of the face region, facial contour coordinate information can be obtained. Then, facial contour fitting processing is performed on the target face based on the facial contour coordinate information to obtain a facial contour outline. Finally, the facial contour outline is mapped onto a color image to obtain the final facial contour. Figure 15 The image mapped to the target contour curve is a color image, so that subsequent steps can automatically expose the target face in the color image based on the target contour curve, thereby improving the adjustment effect of FaceAE processing.

[0164] Step 1009: Perform automatic face exposure processing on the target face in the color image according to the automatic exposure adjustment parameters.

[0165] In this step, when the automatic exposure adjustment parameter corresponds to the adjustment parameter of each pixel within the target contour curve frame, when performing automatic face exposure processing on the target face in the color image according to the automatic exposure adjustment parameter, the brightness of each pixel within the target contour curve frame can be adjusted according to the automatic exposure adjustment parameter, reducing the brightness of pixels with higher original brightness values ​​and increasing the brightness of pixels with lower original brightness values. When the automatic exposure adjustment parameter corresponds to the adjustment parameter of each pixel block divided according to brightness statistics, when performing automatic face exposure processing on the target face in the color image according to the automatic exposure adjustment parameter, an exposure weight value can be set for each pixel block according to the automatic exposure adjustment parameter, and then the brightness of each pixel block can be adjusted according to the exposure weight value, reducing the overall brightness of pixel blocks with lower exposure weight values ​​and increasing the overall brightness of pixel blocks with higher exposure weight values.

[0166] Step 1010: Output the target image after automatic face exposure processing.

[0167] In this step, after the automatic exposure processing of the target face in the color image is completed in step 1009, the target image with the automatic exposure processing completed can be output so that subsequent steps can perform operations such as facial recognition payment or facial recognition login based on the target image. For example... Figure 16 As shown, Figure 16 This is a comparison image showing the effect of FaceAE processing on a target image obtained using a rectangular bounding box and FaceAE processing on a target image obtained using an elliptical bounding box. Figure 16 The image on the left is a schematic diagram of the target image obtained by FaceAE processing using a rectangular bounding box. Figure 16 The image on the right is a schematic diagram of the target image obtained by FaceAE processing using an elliptic curve bounding box. According to... Figure 16 The comparison between the target images shows that the target image obtained by FaceAE processing using a rectangular frame has an overall darker effect, and the boundary between the dark edge of the face and the background environment is not obvious, indicating a poor adjustment effect of FaceAE processing. On the other hand, the target image obtained by FaceAE processing using an elliptical curve frame has an overall brighter effect, and the boundary between the face area and the background environment is obvious, indicating a better adjustment effect of FaceAE processing.

[0168] Step 1011: End processing.

[0169] In this embodiment, using the image processing method including steps 1001 to 1011, after acquiring the pixel-aligned infrared image, depth image, and color image of the target face, face detection processing is first performed on the infrared image to obtain the face detection bounding box of the target face. Then, the coordinate information of the face detection bounding box in the infrared image is mapped to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction processing is performed on the depth image based on the target coordinate information to obtain the face region depth map of the target face. Finally, face contour fitting processing is performed on the target face in the face region depth map to obtain the face contour curve box. Since the infrared image and depth image... Both methods are unaffected by ambient light intensity, and the depth image can reflect the geometric shape information of the target face. Therefore, when uneven ambient light causes no clear boundary between the face region and the background region in the color image, the infrared image can obtain a more accurate face detection bounding box. Then, based on the face detection bounding box and the depth image, a depth map of the target face region can be effectively obtained. Subsequently, based on the depth map of the face region, a more accurate face contour curve box of the target face can be obtained, thereby improving the accuracy of face region detection. This more accurate face contour curve box can then be used to improve the adjustment effect of FaceAE processing for color images. Furthermore, during the execution of steps 1001 to 1011, no higher computing power is required from the corresponding hardware devices, thus improving the accuracy of face region detection can be achieved with lower hardware costs.

[0170] In some possible implementations, assuming the image to be processed contains more than two faces, different detection methods can be used based on the relative positions of these faces within the image. For example, if the image contains two faces, and one face occupies a much larger proportion of the image than the other, the face with the larger proportion can be identified as the target face, and the preceding image processing methods can be performed. After obtaining the face contour bounding box, FaceAE processing can be performed using this bounding box. Alternatively, if the image contains two faces with similar proportions, both faces can be identified as target faces, and the preceding image processing methods can be performed on each target face. After obtaining the respective face contour bounding boxes, FaceAE processing can be performed on each face contour bounding box. For example, suppose the image to be processed contains two faces, and the proportions of the two faces in the image are not significantly different, but one face occludes the other. In this case, both faces can be identified as target faces. For the unoccluded target face, the image processing method described above is used to obtain the face contour bounding box. For the occluded target face, during the image processing, when detecting face bounding boxes in the infrared image, the occluded part is discarded, and only the corresponding face contour bounding box is obtained for the unoccluded part. After obtaining each face contour bounding box, FaceAE processing is performed based on each face contour bounding box. Furthermore, for the case where the image to be processed contains three or more faces, the detection principle is similar to the principle described above, and will not be repeated here to avoid redundancy.

[0171] The following examples illustrate the application scenarios of the embodiments of the present invention.

[0172] Scene 1

[0173] The image processing method provided in this invention can be applied to facial recognition payment scenarios. Specifically, in response to a facial recognition payment request from a user, a smartphone or cashier acquires an infrared image, a depth image, and a color image containing the target face of the user. There is a correspondence between the pixels in the infrared image, depth image, and color image. Then, face detection is performed on the infrared image to obtain a face detection bounding box for the target face. The coordinate information of the face detection bounding box in the infrared image is mapped to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Then, face contour fitting is performed on the target face in the face region depth map to obtain a face contour curve frame. After obtaining the face contour curve frame, it is mapped to the color image to obtain a target contour curve frame. FaceAE processing is then performed on the target face in the color image based on the target contour curve frame to obtain a target image. Finally, face recognition is performed based on the target image. Once recognition is successful, the payment request is completed.

[0174] Scene 2

[0175] The image processing method provided in this embodiment of the invention can also be applied to access control scenarios. Specifically, in response to a resident's request to open the door, the access control device acquires an infrared image, a depth image, and a color image containing the resident's target face. There is a correspondence between the pixels in the infrared image, the depth image, and the color image. Then, face detection is performed on the infrared image to obtain a face detection bounding box for the target face. The coordinate information of the face detection bounding box in the infrared image is mapped to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Then, face contour fitting is performed on the target face in the face region depth map to obtain a face contour curve frame. After obtaining the face contour curve frame, it is mapped to the color image to obtain a target contour curve frame. FaceAE processing is then performed on the target face in the color image based on the target contour curve frame to obtain a target image. Finally, face recognition is performed based on the target image, and the access control is opened after successful recognition.

[0176] Scene 3

[0177] The image processing method provided in this embodiment of the invention can also be applied to vehicle scenarios. Specifically, in response to the driver's vehicle start request, the vehicle terminal acquires an infrared image, a depth image, and a color image containing the driver's target face. There is a correspondence between the pixels in the infrared image, depth image, and color image. Then, face detection is performed on the infrared image to obtain a face detection bounding box for the target face. The coordinate information of the face detection bounding box in the infrared image is mapped to the depth image to obtain the target coordinate information of the face detection bounding box in the depth image. Next, face region extraction is performed on the depth image based on the target coordinate information to obtain a face region depth map of the target face. Then, face contour fitting is performed on the target face in the face region depth map to obtain a face contour curve frame. After obtaining the face contour curve frame, it is mapped to the color image to obtain a target contour curve frame. FaceAE processing is then performed on the target face in the color image based on the target contour curve frame to obtain a target image. Finally, face recognition is performed based on the target image. After successful recognition, the vehicle is started.

[0178] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0179] Reference Figure 17 The present invention also discloses an image processing apparatus 170, which is capable of implementing the image processing method as described in the preceding embodiments. The image processing apparatus 170 includes:

[0180] Image acquisition unit 171 is used to acquire a target face image to be processed. The image to be processed includes an infrared image and a depth image. There is a correspondence between the pixels of the infrared image and the pixels of the depth image.

[0181] The face detection unit 172 is used to perform face detection on the infrared image and obtain the face detection box of the target face;

[0182] The coordinate mapping unit 173 is used to map the face detection box to the depth image to obtain the target coordinate information of the face detection box in the depth image;

[0183] The face extraction unit 174 is used to extract the face region from the depth image based on the target coordinate information to obtain the face region depth map of the target face.

[0184] Image exposure unit 175 is used to fit the contour of the target face in the depth map of the face region to obtain the face contour curve frame, so as to use the face contour curve frame for subsequent image processing.

[0185] In one embodiment, the face extraction unit 174 is further configured to:

[0186] The depth image is segmented into foreground and background regions to obtain a denoised depth map;

[0187] The face region is extracted from the depth image based on the denoised depth map and the target coordinate information to obtain the face region depth map of the target face.

[0188] In one embodiment, the face extraction unit 174 is further configured to:

[0189] The target face region is extracted from the denoised depth map based on the target coordinate information;

[0190] Based on the target coordinate information, the target face region is copied to the target region in the depth image to obtain the depth map of the target face region.

[0191] In one embodiment, the face extraction unit 174 is further configured to:

[0192] Set all depth pixel values ​​in the depth image to the first target value to obtain a backup depth image;

[0193] Based on the target coordinate information, the target face region is copied to the target region in the backup depth map to obtain the face region depth map of the target face.

[0194] In one embodiment, the face extraction unit 174 is further configured to:

[0195] Obtain the foreground segmentation threshold and background segmentation threshold;

[0196] In the depth image, depth pixel values ​​that are less than the foreground segmentation threshold are set as the second target value, and depth pixel values ​​that are greater than the background segmentation threshold are set as the second target value to obtain a denoised depth map.

[0197] In one embodiment, the face extraction unit 174 is further configured to:

[0198] Obtain the average depth pixel value of the depth image;

[0199] The foreground segmentation threshold is calculated based on the average depth pixel value and the first depth threshold.

[0200] The background segmentation threshold is calculated based on the average depth pixel value and the second depth threshold.

[0201] In one embodiment, the image exposure unit 175 is further configured to:

[0202] Contour extraction is performed on the target face in the depth map of the face region to obtain the face contour coordinate information;

[0203] The contour of the target face is fitted based on the face contour coordinate information to obtain the face contour curve frame.

[0204] In one embodiment, the image exposure unit 175 is further configured to:

[0205] Binarization is performed on the depth map of the face region to obtain a binarized depth map;

[0206] The binary depth map is converted to a different data type to obtain the target depth map;

[0207] The contour of the target face in the depth map is extracted to obtain the face contour coordinate information.

[0208] In one embodiment, the image to be processed further includes a color image, wherein the pixels of the color image and the pixels of the depth image have a corresponding relationship; the image exposure unit 175 is further configured to:

[0209] The facial contour outline is mapped onto a color image to obtain the target contour outline.

[0210] Automatic face exposure is performed on the target face in the color image based on the target contour curve to obtain the target image.

[0211] In one embodiment, the image exposure unit 175 is further configured to:

[0212] Brightness statistics are obtained by statistically analyzing the brightness information of the target face in the color image based on the target contour curve.

[0213] Automatic face exposure is performed on the target face in the color image based on brightness statistics to obtain the target image.

[0214] In one embodiment, the image exposure unit 175 is further configured to:

[0215] Automatic exposure adjustment parameters are calculated based on brightness statistics.

[0216] The target face in the color image is automatically exposed according to the automatic exposure adjustment parameters to obtain the target image.

[0217] It should be noted that since the image processing device 170 of this embodiment can implement the image processing method described in the previous embodiment, the image processing device 170 of this embodiment has the same technical principle and the same beneficial effect as the image processing method described in the previous embodiment. To avoid repetition, it will not be described again here.

[0218] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0219] Reference Figure 18 The present invention also discloses an image processing apparatus, the image processing apparatus 180 comprising:

[0220] At least one processor 181;

[0221] At least one memory 182 is used to store at least one program;

[0222] When at least one program is executed by at least one processor 181, the image processing method as described in any of the preceding embodiments is implemented.

[0223] This invention also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the image processing method as described in any of the preceding embodiments.

[0224] This invention also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the image processing method as described in any of the preceding embodiments.

[0225] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0226] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0227] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0230] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0231] The step numbers in the above method embodiments are set only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. An image processing method, characterized in that, Includes the following steps: A target face image to be processed is acquired. The image to be processed includes a color image, an infrared image, and a depth image. The pixels of the color image, the infrared image, and the depth image are completely aligned, and the color image, the infrared image, and the depth image are time-synchronized. Face detection is performed on the infrared image to obtain a face detection box of the target face, wherein the face detection box is a rectangular detection box that matches the contour of the target face; The face detection box is mapped onto the depth image to obtain the target coordinate information of the face detection box in the depth image; Obtain the average depth pixel value of the depth image; The first and second depth thresholds are determined based on prior knowledge of the head thickness value. The foreground segmentation threshold is calculated based on the mean depth pixel value and the first depth threshold. The background segmentation threshold is calculated based on the mean depth pixel value and the second depth threshold. In the depth image, depth pixel values ​​smaller than the foreground segmentation threshold are set as the second target value, and depth pixel values ​​larger than the background segmentation threshold are set as the second target value to obtain a denoised depth map; The target face region is extracted from the denoised depth map based on the target coordinate information. The target face region is copied into the target region of the depth image based on the target coordinate information to obtain the face region depth map of the target face; The depth map of the face region is binarized to obtain a binarized depth map; The binarized depth map is converted to a new data type to obtain the target depth map. The target face in the target depth map is subjected to face contour extraction processing to obtain face contour coordinate information; Based on the facial contour coordinate information, the target face is fitted with a contour to obtain a facial contour curve that matches the contour of the target face. The face contour curve is mapped onto the color image to obtain the target contour curve. Automatic face exposure is performed on the target face in the color image based on the target contour curve to obtain the target image.

2. The image processing method according to claim 1, characterized in that, The step of copying the target face region to the target region in the depth image based on the target coordinate information to obtain the face region depth map of the target face includes: Set all depth pixel values ​​in the depth image to the first target value to obtain a backup depth map; The target face region is copied to the target region in the backup depth map based on the target coordinate information to obtain the face region depth map of the target face.

3. The image processing method according to claim 1, characterized in that, The step of automatically exposing the target face in the color image based on the target contour curve to obtain the target image includes: Based on the target contour curve frame, the brightness information of the target face in the color image is statistically analyzed to obtain brightness statistical information; Based on the brightness statistics, the target face in the color image is automatically exposed to obtain the target image.

4. The image processing method according to claim 3, characterized in that, The step of automatically exposing the target face in the color image based on the brightness statistics to obtain the target image includes: The automatic exposure adjustment parameters are calculated based on the brightness statistics. The target face in the color image is automatically exposed according to the automatic exposure adjustment parameters to obtain the target image.

5. An image processing apparatus, characterized in that, include: An image acquisition unit is used to acquire a target face image to be processed. The image to be processed includes a color image, an infrared image, and a depth image. The pixels of the color image, the infrared image, and the depth image are completely aligned, and the color image, the infrared image, and the depth image are time-synchronized. A face detection unit is used to perform face detection on the infrared image to obtain a face detection box of the target face, wherein the face detection box is a rectangular detection box that matches the contour of the target face. A coordinate mapping unit is used to map the face detection box to the depth image to obtain the target coordinate information of the face detection box in the depth image; A face extraction unit is used to obtain the average depth pixel value of the depth image; A first depth threshold and a second depth threshold are determined based on prior knowledge of the head thickness value; a foreground segmentation threshold is calculated based on the mean depth pixel value and the first depth threshold; a background segmentation threshold is calculated based on the mean depth pixel value and the second depth threshold. In the depth image, depth pixel values ​​smaller than the foreground segmentation threshold are set as the second target value, and depth pixel values ​​larger than the background segmentation threshold are set as the second target value to obtain a denoised depth map; the target face region is cropped in the denoised depth map according to the target coordinate information; the target face region is copied to the target region in the depth image according to the target coordinate information to obtain the face region depth map of the target face; The image exposure unit is configured to perform binarization processing on the depth map of the face region to obtain a binarized depth map, perform data type conversion processing on the binarized depth map to obtain a target depth map, perform face contour extraction processing on the target face in the target depth map to obtain face contour coordinate information, perform contour fitting on the target face according to the face contour coordinate information to obtain a face contour curve frame that matches the contour of the target face, map the face contour curve frame onto the color image to obtain a target contour curve frame, and perform automatic face exposure on the target face in the color image according to the target contour curve frame to obtain a target image.

6. An image processing apparatus, characterized in that, include: At least one processor; At least one memory for storing at least one program; The image processing method as described in any one of claims 1 to 4 is implemented when at least one of the programs is executed by at least one of the processors.

7. A computer-readable storage medium, characterized in that, It stores a processor-executable program, which, when executed by a processor, is used to implement the image processing method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, causing the computer device to perform the image processing method as described in any one of claims 1 to 4.

Images