Face image processing method and device, electronic equipment and storage medium

Abstract

The invention provides a face image processing method and device, electronic equipment and a storage medium. The method comprises the steps of calling a first auxiliary module based on a first sample face image, and determining first loss; calling a second auxiliary module based on second sample face images of different definition types, and determining second loss; updating parameters of the image processing model based on the first loss and the second loss to obtain a trained image processing model; calling the trained image processing model based on the video frame image to perform feature extraction to obtain a first image feature; and calling a trained image processing model to predict a definition index of the video frame image based on the first image feature. Through the method and the device, the accuracy of definition recognition for the face image can be improved, and then the accuracy of face recognition is improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, electronic device and storage medium for processing facial images. Background Technology

[0002] Facial recognition is a technology that identifies individuals based on their facial biometric features. It involves acquiring images or video streams containing faces, using specialized algorithms to detect and locate facial regions, extracting biometric information from the face and constructing a feature model, and then comparing this feature model with known facial feature models in a database to complete identity verification or retrieval.

[0003] In facial recognition, image sharpness is one of the core factors determining recognition accuracy. A sharp facial image can fully preserve key biometric details, providing sufficient and effective information for feature extraction algorithms and ensuring the extracted feature model has high discriminative power. Simultaneously, high sharpness reduces noise interference during feature extraction, improving the matching accuracy and reliability during feature comparison, and avoiding feature overlap or distortion caused by image blur, thereby reducing the risk of misidentification or recognition failure. The low accuracy of blur-based judgment algorithms in related technologies consequently affects the accuracy of facial recognition. Summary of the Invention

[0004] This application provides a face image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of face image clarity recognition, thereby improving the accuracy of face recognition.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a face image processing method, the method comprising: Based on the first sample face image, the first auxiliary module is invoked to perform feature extraction to obtain the first predicted image; The first loss is determined based on the difference between the first predicted image and the first facial feature extraction image; Based on the second sample face images of different resolution types, the second auxiliary module is called to extract features and obtain second predicted images of different resolution types; The second loss is determined based on the difference between the second predicted image of different resolution types and the corresponding second facial feature extraction image. The parameters of the image processing model are updated based on the first loss and the second loss to obtain the trained image processing model. Based on the video frame images, the trained image processing model is invoked to extract features and obtain the first image features; Based on the first image features, the trained image processing model is invoked to predict the sharpness index of the video frame image.

[0006] This application provides a face image processing device, including: The loss acquisition module is used to call the first auxiliary module to perform feature extraction based on the first sample face image to obtain the first predicted image; determine the first loss based on the difference between the first predicted image and the first facial feature extraction image; call the second auxiliary module to perform feature extraction based on the second sample face images of different resolution types to obtain the second predicted images of different resolution types; determine the second loss based on the difference between the second predicted images of different resolution types and the corresponding second facial feature extraction images. The training module is used to update the parameters of the image processing model based on the first loss and the second loss to obtain the trained image processing model. The image processing module is used to extract features based on the trained image processing model using video frame images to obtain the first image features; The image processing module is further configured to, based on the first image features, invoke the trained image processing model to predict the sharpness index of the video frame image.

[0007] In some embodiments, the image processing module is configured to invoke the trained image processing model based on the first image features to perform probability prediction processing, thereby obtaining a first predicted probability that the video frame image belongs to different sharpness types; obtain a preset value corresponding to each sharpness type; and perform a linear operation based on the first predicted probability of each sharpness type and the preset value to obtain the sharpness index of the video frame image.

[0008] In some embodiments, the image processing module is configured to multiply the first predicted probability by the preset value for each of the sharpness types to obtain a first product; and to use the sum of each first product as the sharpness index of the video frame image.

[0009] In some embodiments, the loss acquisition module is configured to, before determining the first loss based on the difference between the first predicted image and the first facial feature extraction image, perform facial feature segmentation on the first sample face image to obtain the first facial feature extraction image by: performing keypoint recognition on the first sample face image to obtain keypoints corresponding to the facial features in the first sample face image; connecting the keypoints of each part of the facial features to obtain a first contour line of each part; performing curve processing on each first contour line to obtain a second contour line; taking the area surrounded by the second contour line in the first sample face image as the facial feature region; constructing a facial feature heatmap based on the facial feature region, wherein the pixel value outside the facial feature region in the facial feature heatmap is 0; and performing matrix multiplication on the first sample face image and the facial feature heatmap to obtain the first facial feature extraction image.

[0010] In some embodiments, the loss acquisition module is configured to: determine the second pixel corresponding to each first pixel in the first predicted image in the first facial feature extraction image; determine a first difference between the first pixel value of each first pixel and the second pixel value of the corresponding second pixel; determine the square of each first difference; and take the average of each square as the first loss.

[0011] In some embodiments, the loss acquisition module is used to perform noise processing on the first sample face image to different degrees before calling the second auxiliary module to perform feature extraction based on the second sample face image of different resolution types to obtain the second predicted image of different resolution types; before determining the second loss based on the difference between the second predicted image of different resolution types and the corresponding second facial feature extraction image, the method further includes: performing facial feature segmentation on the second sample face image of different resolution types to obtain the corresponding second facial feature extraction image.

[0012] In some embodiments, the image processing module is configured to extract and process the video frame image to obtain a second image feature; compress the second image feature to obtain a third image feature with a lower dimension than the second image feature; and activate the third image feature to obtain the first image feature.

[0013] In some embodiments, the image processing module is configured to, after predicting the sharpness index of the video frame image by calling the trained image processing model based on the first image features, discard the video frame image when the sharpness index of the video frame image is less than a preset threshold; and perform identity recognition on the face in the video frame image when the sharpness index of the video frame image is greater than or equal to the preset threshold.

[0014] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the face image processing method provided in the embodiments of this application.

[0015] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the face image processing method provided in this application.

[0016] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the face image processing method provided in this application.

[0017] The embodiments of this application have the following beneficial effects: By constructing a first auxiliary module and a second auxiliary module outside the image processing model, the loss between the first predicted image and the first facial feature extraction image can be obtained, thereby improving the sensitivity of the image processing model to the face region. The image processing model can determine the facial feature details in the face image and then judge the sharpness based on the features of the facial feature details. It can also obtain the loss between the second predicted image of different sharpness types and the corresponding second facial feature extraction image, thereby improving the sensitivity of the image processing model to different sharpness of the face region. The image processing model can determine the facial feature details in face images of different sharpness and then judge the sharpness based on the features of the facial feature details. In the application stage, the first auxiliary module and the second auxiliary module do not participate in the calculation process of the image processing model, saving the storage space occupied by the image processing model and improving the computational efficiency and the accuracy of sharpness prediction. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the application mode of the face image processing method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3A This is a first flowchart illustrating the face image processing method provided in this application embodiment; Figure 3B This is a schematic diagram of the second process of the face image processing method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the third process of the face image processing method provided in the embodiments of this application; Figure 5AThis is a first structural schematic diagram of the image processing model provided in the embodiments of this application; Figure 5B This is a schematic diagram of the second structure of the image processing model provided in the embodiments of this application; Figure 5C This is a schematic diagram of the third structure of the image processing model provided in the embodiments of this application; Figure 6A This is a fourth structural schematic diagram of the image processing model provided in the embodiments of this application; Figure 6B This is a fifth structural schematic diagram of the image processing model provided in the embodiments of this application; Figure 7A This is a schematic diagram of facial key points provided in an embodiment of this application; Figure 7B This is a schematic diagram of the heat map of the five senses provided in the embodiments of this application; Figure 7C This is a schematic diagram of facial feature heatmaps with different resolutions provided in the embodiments of this application.

[0019] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0022] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] In this application, the facial (or other biometric) recognition technology involved, when applied to specific products or technologies in the above embodiments of this application, the relevant data collection, use and processing processes should comply with the requirements of relevant laws and regulations. Before collecting facial information, the information processing rules should be communicated and the individual consent of the target should be obtained. Facial information should be processed in strict accordance with the requirements of laws and regulations and personal information processing rules, and technical measures should be taken to ensure the security of relevant data.

[0024] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0027] 1) Facial recognition: This is a biometric technology that identifies individuals based on their facial features. It involves using cameras or webcams to capture images or video streams containing faces, automatically detecting faces in the images, and then performing facial recognition on the detected faces. It is also commonly known as portrait recognition or face recognition.

[0028] 2) Visual Geometry Group Network (VGG): A deep convolutional neural network architecture specifically designed for image recognition tasks. The core feature of VGG is the use of small convolutional kernels (3×3) and pooling layers to construct a deep network, significantly improving image classification accuracy by increasing network depth.

[0029] 3) Structural re-parameterization: This refers to first constructing a series of structures (generally used for training), and then converting their parameters into another set of parameters (generally used for inference), thereby converting this series of structures into another series of structures.

[0030] 4) Reparameterized Visual Geometry Group Network (RepVGG): This is a deep learning model that combines reparameterization technology with the characteristics of Visual Geometry Group Network (VGG), aiming to improve performance and efficiency through structural optimization.

[0031] 5) Facial landmark detection: This refers to the technology of locating the key areas of a face given a face image, including eyebrows, eyes, nose, mouth, and facial contours.

[0032] 6) U-Net: A medical image segmentation model based on Convolutional Neural Networks (CNNs). The core feature of U-Nets is the U-shaped encoder-decoder structure, which achieves multi-scale feature fusion through skip connections. It has been widely applied in fields such as autonomous driving and medical image analysis.

[0033] 7) Heatmap: A heatmap is a visualization tool that uses color depth or point density distribution to show the spatial distribution and density differences of data. It is widely used in geographic information, market analysis, user behavior research, and other fields. In this embodiment, the facial feature heatmap quantifies and displays the location, salience, and information contribution of key biometric features of the five senses through the spatial distribution of color depth, pixel weight, or local feature density, providing intuitive technical support for facial feature region detection, feature extraction effectiveness verification, and recognition model optimization.

[0034] In facial recognition, image sharpness is one of the core factors determining recognition accuracy. A sharp facial image can completely preserve key biometric details, providing sufficient and effective information for feature extraction algorithms and ensuring the extracted feature model has high discriminative power. Simultaneously, high sharpness reduces noise interference during feature extraction, improving the matching accuracy and reliability during feature comparison, and avoiding feature overlap or distortion caused by image blur, thereby reducing the risk of misidentification or recognition failure. Related blur-based judgment algorithms are all based on image gradients; however, when there is ambient light interference, the gradient information used by the gradient function is destroyed by the ambient light, causing interference. Image gradient-based judgment algorithms have poor anti-interference capabilities and low accuracy, affecting the accuracy of facial recognition.

[0035] This application provides a face image processing method, a face image processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of face image clarity recognition, thereby improving the accuracy of face recognition.

[0036] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as terminal devices, such as laptops, tablets, desktop computers, set-top boxes, smart TVs, in-vehicle terminals, virtual reality (VR) devices, augmented reality (AR) devices, and other various types of terminals. They can also be implemented as servers. The following will describe exemplary applications when the electronic device is implemented as a terminal device or a server.

[0037] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the face image processing method provided in the embodiments of this application; Figure 1 It includes terminal device 400, server 200, network 300, and database 500. Terminal device 400 is connected to server 200 through network 300, which can be a wide area network, a local area network, or a combination of both.

[0038] Terminal device 400 includes a camera that can be used to capture images containing human faces, server 200 can be a server for a face recognition system, and database 500 can store a large number of faces and their corresponding identity information.

[0039] For example, the terminal device 400 calls the face image processing method provided in the embodiment of this application to train an image processing model. When a person appears in the field of view of the camera of the terminal device 400, the camera captures an image. The terminal device 400 calls the image processing model trained by the face image processing method provided in the embodiment of this application to perform sharpness detection on the captured image and obtain the image sharpness index. When the image sharpness index is greater than a preset threshold, face recognition is performed based on the captured image, and the corresponding identity information is matched.

[0040] In some embodiments, the face image processing method of this application can also be applied in the following application scenarios: (1) Face recognition of access control system: The camera of the terminal device of access control system collects images within the field of view, calls the image processing model trained by the face image processing method provided in this application embodiment, performs clarity detection on the collected images, and obtains the image clarity index. When the image clarity index is greater than the preset threshold, face recognition is performed based on the collected images and the corresponding identity information is matched. If the identity information is a user with permission to access, the door is opened.

[0041] (2) Face recognition of the welcoming system: The camera of the terminal device of the welcoming system collects images within the field of view, calls the image processing model trained by the face image processing method provided in the embodiment of this application, performs clarity detection on the collected images, and obtains the image clarity index. When the image clarity index is greater than the preset threshold, face recognition is performed based on the collected images, and the corresponding identity information is matched. If the identity information is a user with permission to access, the corresponding welcoming operation is performed.

[0042] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0043] This application embodiment can be implemented using database technology. A database, simply put, can be viewed as an electronic filing cabinet storing electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, capable of being shared by multiple users, having minimal redundancy, and being independent of application programs.

[0044] A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile devices; or according to the query language used, such as Structured Query Language (SQL) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0045] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be... Figure 1 Terminal device 400, Figure 2 The terminal device 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0046] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0047] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0048] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0049] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0050] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0051] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430. The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0052] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A face image processing apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plug-ins, including the following software modules: a loss acquisition module 4551, a training module 4552, and an image processing module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. Figure 2 For ease of explanation, all the above modules are shown at once, and the functions of each module will be explained below.

[0053] In other embodiments, the face image processing apparatus provided in this application can be implemented in hardware. As an example, the face image processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the face image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0054] The face image processing method provided in this application will be described in conjunction with exemplary applications and implementations of the terminal devices provided in the embodiments of this application.

[0055] The following describes the face image processing method provided in the embodiments of this application. As mentioned above, the electronic device implementing the face image processing method of the embodiments of this application can be a terminal device or a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0056] It should be noted that the examples of face image processing in the following text are illustrated using an application scenario of identity recognition. Based on the understanding of the following text, those skilled in the art can apply the face image processing method provided in the embodiments of this application to the processing of clarity detection, including other types of target recognition.

[0057] See Figure 3A , Figure 3A This is a flowchart illustrating the face image processing method provided in the embodiments of this application, which will be combined with... Figure 3A The steps shown are explained. Figure 3A The entity responsible for executing the steps is Figure 1 400 terminal devices.

[0058] In step 301, the first auxiliary module is invoked to extract features based on the first sample face image to obtain the first predicted image.

[0059] For example, the first auxiliary module (or auxiliary branch) and the second auxiliary module mentioned below are network structures independent of the image processing model and do not participate in the image processing process of the image processing model. The first auxiliary module can be a network structure with image feature extraction capabilities, such as a U-shaped network (Unet). Feature extraction based on the first sample face image can be achieved by calling the first auxiliary module in the following way: perform multiple levels of upsampling operations on the first sample face image, with the input of the feature extraction processing at each level being the output of the previous level, to obtain the first predicted image. During the upsampling process, the number of feature channels is adjusted by 1×1 convolution, and then concatenated with the low-level feature map of the same scale as the encoder to compensate for the loss of details caused by upsampling; the fused feature map is further optimized by 3×3 convolution, integrating the high-level semantic features and low-level detail features of the face to achieve accurate representation of local features (such as facial features) and global structural features of the face. The output layer maps the feature map output by the decoder to the target number of channels (depending on the task requirements) through 1×1 convolution, and generates the first predicted image after processing by the activation function.

[0060] In some embodiments, reference Figure 6A , Figure 6AThis is a schematic diagram of the fourth structure of the image processing model provided in this application embodiment; the structure of the first auxiliary branch refers to the U-shaped network (UNet) structure, uses a 3x3 auxiliary convolutional layer (conv_aux) to extract features, concatenates them with the features of the previous level, and then adjusts the dimensions through a 3x3 convolutional layer (conv_aux), and so on to obtain the final first predicted image.

[0061] In some embodiments, before step 302, the first sample face image is segmented into facial features in the following manner to obtain the first facial feature extraction image: Key point recognition is performed on the first sample face image to obtain the key points corresponding to the facial features in the first sample face image; the key points of each part of the facial features are connected to obtain the first contour line of each part; each first contour line is processed into a curve to obtain the second contour line; the area surrounded by the second contour line in the first sample face image is taken as the facial feature region; a facial feature heat map is constructed based on the facial feature region, wherein the pixel value outside the facial feature region in the facial feature heat map is 0; the first sample face image and the facial feature heat map are matrix multiplied to obtain the first facial feature extraction image.

[0062] For example, a lightweight convolutional neural network (such as MobileNet or ShuffleNet) or a brute-force transformer model is used to extract hierarchical features of the face region (from low-level features such as edges and textures to high-level semantic features such as facial structure and pose). The pixel coordinates of keypoints are directly predicted through a regression branch, and the pixel coordinates of the keypoints of each part of the facial features (eyes, nose, mouth) are connected to form a first contour line. A geometric fitting algorithm transforms the first contour line into a continuous and smooth second contour line, which is a closed boundary. The second contour line must conform to the natural shape of the facial features (e.g., eyes are ellipses, eyebrows are arcs, and the mouth is a closed curve). The area enclosed by the closed boundary is taken as the facial feature region, and the pixels with coordinates within this region are taken as the pixels of the facial feature region. The facial feature region is represented by 1 (white, highlight), and the background by 0 (black, low brightness), i.e., a binary heatmap. (Reference) Figure 7A , Figure 7A This is a schematic diagram of facial key points provided in an embodiment of this application; see reference. Figure 7B , Figure 7B This is a schematic diagram of the facial feature heatmap provided in this embodiment. The sample face image 701 is multiplied by the facial feature heatmap 702. Since the parameter in the facial feature heatmap is 0, the result does not contain black areas, but only the face sample image corresponding to the white areas of the facial feature heatmap 702, i.e., the facial features. Based on the facial feature image, the image processing model described above is used to predict the prediction probability of the first face image for different resolution types.

[0063] In this embodiment, the location of facial features is determined by facial key point recognition, and the facial feature images are extracted by facial feature heatmap, which can improve the accuracy of facial feature image extraction.

[0064] In step 302, a first loss is determined based on the difference between the first predicted image and the first facial feature extraction image.

[0065] For example, the difference between the first predicted image and the first facial feature extraction image is the difference in pixel values ​​of the corresponding pixels (the difference in pixel values ​​or the square of the difference in pixel values). The first loss can be the average of the pixel value differences.

[0066] In some embodiments, step 302 can be implemented by: determining the second pixel corresponding to each first pixel in the first predicted image in the first facial feature extraction image; determining a first difference between the first pixel value of each first pixel and the second pixel value of the corresponding second pixel; determining the square of each first difference; and taking the average of each square as a first loss.

[0067] The first loss could be the mean squared error loss. It is expressed as the following formula (2): (2) Where P is the pixel value of the first predicted image, T is the pixel value of the extracted facial features image, and H×W is the image size. ) is the first difference, This represents the average value obtained by summing the squares and dividing by the total number of pixels.

[0068] In some embodiments, the following process is performed before step 303: the first sample face image is subjected to different degrees of noise addition to obtain second sample face images of different clarity types.

[0069] For example, during the noise addition process, the noise type and intensity are controlled by parameterization. Each sharpness type corresponds to a different noise intensity. A fixed random seed (such as np.random.seed(42)) is set to ensure that the noise addition results of different face images can be reproduced under the same parameters, and the noise intensity of the same sharpness type is consistent. The second sample face image with the highest sharpness is the original first sample face image.

[0070] In step 303, the second auxiliary module is invoked to extract features based on the second sample face images of different resolution types to obtain second predicted images of different resolution types.

[0071] For example, the structure of the second auxiliary module is the same as that of the first auxiliary module, see reference. Figure 6B , Figure 6BThis is a fifth structural schematic diagram of the image processing model provided in this application embodiment; to distinguish between "relatively clear" and "clear," the relatively clear heatmap is blurred and noise-added to reflect that only blurred facial features are visible in the relatively clear image. (Reference) Figure 7C , Figure 7C This is a schematic diagram of facial feature heatmaps with different resolutions provided in the embodiments of this application. For a clear image, the second auxiliary module is used to predict a clear facial feature heatmap; for a relatively clear image, the second auxiliary module is used to predict a relatively clear facial feature heatmap with added noise; and for a blurry image, a blurry heatmap cannot be predicted.

[0072] In some embodiments, the following processing is performed before step 304: facial feature segmentation is performed on the second sample face images of different resolution types to obtain the corresponding second facial feature extraction images.

[0073] For example, the principle of facial feature segmentation is explained above, and will not be repeated here.

[0074] In step 304, a second loss is determined based on the difference between the second predicted image of different resolution types and the corresponding second facial feature extraction image.

[0075] For example, the second loss can also be the mean squared error loss. For the specific principle, please refer to step 302 above, which will not be repeated here.

[0076] In step 305, the parameters of the image processing model are updated based on the first loss and the second loss to obtain the trained image processing model.

[0077] For example, the first loss and the second loss are added together to obtain the total loss. Based on the total loss, the image processing model is backpropagated, and the parameters of the image processing model are adjusted in the opposite direction of the gradient to minimize the total loss, thus obtaining the trained image processing model.

[0078] In step 306, the trained image processing model is invoked based on the video frame image to perform feature extraction, thereby obtaining the first image features.

[0079] For example, the image processing model could be a Reparameterized Visual Geometry Group Network (RepVGG), a deep learning model that combines reparameterization techniques with the characteristics of Visual Geometry Group Networks (VGG). A video frame image is an image containing a face that is to be identified. A video frame image is a specific frame extracted from a video captured by a camera.

[0080] In some embodiments, step 306 can be implemented by: extracting and processing the video frame image to obtain a second image feature; compressing the second image feature to obtain a third image feature with a lower dimension than the second image feature; and activating the third image feature to obtain a first image feature.

[0081] refer to Figure 5A , Figure 5A This is a first structural diagram of the image processing model provided in this application embodiment; the RepVGGA0 model has a structure similar to the Residual Network 501 (ResNet), and is used to extract features from the input image at multiple levels. The RepVGG training phase 502 refers to the RepVGGA0 model during the training phase, and the RepVGG inference phase 503 refers to the RepVGGA0 model during the application phase. The RepVGGA0 model contains multiple convolutional layers and multiple Rectified Linear Unit (ReLU) layers.

[0082] refer to Figure 5B , Figure 5B This is a schematic diagram of the second structure of the image processing model provided in the embodiments of this application; Figure 5B Structure 504 is a component of the RepVGGA0 model. The RepVGGA0 model consists of multiple structures 504, including: a linear rectified function 5041, convolutional layers 5043 and 5042, a batch normalization layer 5044, an additive layer 5046, and an additive layer 5045. Convolutional layers 5043 and 5042 have different sizes. The features output from the linear rectified function 5041 are input into convolutional layers 5043, 5042, and 5044 for processing. The processing results from each layer are summed to obtain the output features.

[0083] In some embodiments, the inference resolution is set to 112. Based on the RepVGGA0-112 classification network, considering the correlation between the features extracted from the deep semantic layer channels and facial features, a channel attention mechanism is introduced into the deep features for recognition. This mechanism is the Squeeze-and-Excitation Network (SENet). The channel attention mechanism module highlights the role of core channels by modeling channel relationships. (Refer to...) Figure 5C , Figure 5C This is a third structural schematic diagram of the image processing model provided in the embodiments of this application, used to characterize the principle of the channel attention mechanism; wherein, It is the basic feature map (Transform Feature), also known as the second image feature. Yes The third image feature obtained by performing a squeeze operation. Yes Obtained through activation. The result (first image feature) is obtained through a channel attention mechanism. Since the core deep semantic channels generally reflect facial details, highlighting the learning of facial feature details can obviously help with facial gender recognition tasks. Compared to adding an attention mechanism to each stage, adding an attention mechanism only to the deep semantic layer can better help with recognition.

[0084] In step 307, based on the first image features, the trained image processing model is invoked to predict the sharpness index of the video frame image.

[0085] For example, the sharpness index can be represented as a number. The trained image processing model can be called to predict the probability that a video frame image belongs to different sharpness types. The predicted probability is then linearly calculated with a preset value to obtain the result.

[0086] In some embodiments, reference Figure 3B , Figure 3A Step 307 can be achieved through Figure 3B Steps 3071 to 3073 are implemented, and the details are explained below.

[0087] In step 3071, the trained image processing model is invoked based on the first image features to perform probability prediction processing, thereby obtaining the first predicted probability that the video frame image belongs to different resolution types.

[0088] For example, the trained image processing model includes an activation function that maps the first image features to predicted probabilities in the (0,1) interval. The mapping is performed separately for each resolution type to obtain the first predicted probability that the video frame image belongs to different resolution types.

[0089] In step 3072, the preset value corresponding to each resolution type is obtained.

[0090] For example, the preset values ​​can be set according to the actual application scenario. In this embodiment, there are three types of clarity (blurry, relatively clear, and clear), and the corresponding preset values ​​are 0, 60, and 100, respectively.

[0091] In step 3073, a linear operation is performed based on the first predicted probability of each sharpness type and a preset value to obtain the sharpness index of the video frame image.

[0092] For example, a linear operation could be to represent the first predicted probability of each sharpness type as a vector, represent the corresponding preset value as a vector, and multiply the two vectors to obtain the sharpness index.

[0093] In some embodiments, step 3073 can be implemented by multiplying the first predicted probability by a preset value for each sharpness type to obtain a first product; and using the sum of each first product as a sharpness index for the video frame image.

[0094] Continuing with the example above, we divide the sharpness type into three categories. When processing sharpness, we adopt the idea of ​​Digital Enhancement for Extreme Clarity (DEX). Assume that y = {0, 60, 100} is assigned, that is, the preset value corresponding to blurriness is 0 points, the preset value corresponding to relatively sharpness is 60 points, and the preset value corresponding to sharpness is 100 points. The result obtained by the neural network is the probability o = {o1, o2, o3} of these three categories, which is used to calculate the final sharpness index E of this image. The calculation formula of sharpness index E is (1): (1) In some embodiments, after step 307, the following processing is performed: when the clarity index of the video frame image is less than a preset threshold, the video frame image is discarded; when the clarity index of the video frame image is greater than or equal to the preset threshold, the face in the video frame image is identified.

[0095] For example, the preset threshold can be set according to the actual application scenario. For example, if the clarity index is greater than 80, it means that the video frame image is relatively clear and can be used for face recognition. Conversely, if the video frame image is relatively blurry, it is not recommended to perform face recognition based on blurry images to avoid recognition errors and waste of computing resources.

[0096] In this embodiment, by constructing a first auxiliary module and a second auxiliary module outside the image processing model, the loss between the first predicted image and the first facial feature extraction image can be obtained, thereby improving the sensitivity of the image processing model to the face region. The image processing model can determine the facial feature details in the face image and then judge the sharpness based on the features of the facial feature details. It can also obtain the loss between the second predicted image of different sharpness types and the corresponding second facial feature extraction image, thereby improving the sensitivity of the image processing model to different sharpness of the face region. The image processing model can determine the facial feature details in face images of different sharpness and then judge the sharpness based on the features of the facial feature details. In the application stage, the first auxiliary module and the second auxiliary module do not participate in the calculation process of the image processing model, saving the storage space occupied by the image processing model and improving the calculation efficiency and the accuracy of sharpness prediction.

[0097] The following will describe an exemplary application of the face image processing method of this application in a real-world application scenario.

[0098] Face quality is a crucial Artificial Intelligence (AI) assisted function in facial recognition greeting features. The goal is to filter high-quality facial data from video trajectories for recognition. During a user's walk, the images captured by the camera may contain occasional low-quality issues (such as a chin obscuring the face due to a momentary head tilt, overexposure due to light reflection, or motion blur), failing to represent the visitor's true facial features. Therefore, it's necessary to filter for clearer faces based on the face quality in the video frames. Common blur detection algorithms are based on image gradients, such as the Brenner gradient function, Tenengrad gradient function, and Laplacian gradient function. Sharpness evaluation methods based on image edge information all measure image sharpness by calculating pixel gradient differences. Image gradient-based algorithms often achieve accurate recognition in simple environments, and the gradient values ​​calculated by these algorithms can characterize the image's sharpness. However, when there is external ambient light interference, the gradient information used by the gradient function is corrupted by the ambient light, causing interference. For example, using Laplace variance analysis, the presence of bright spots in the environment reduces facial sharpness. While using deep learning solutions can bring significant improvements, it can also lead to some misjudgments. Among these, face quality is a relatively subjective factor, and different people may make different judgments.

[0099] To address the aforementioned problems in related technologies, this application provides a face image processing method that employs a lightweight image recognition network and incorporates a channel attention mechanism into deep features for recognition, effectively improving recognition performance. Furthermore, to further emphasize the correlation between extracted features and facial features, a facial feature reconstruction module is introduced for assisted learning, helping the network focus on extracting facial feature clarity, thus enabling face quality screening to provide a foundation for face recognition.

[0100] See Figure 4 , Figure 4 This is a flowchart illustrating the face image processing method provided in the embodiments of this application, which will be combined with... Figure 4 The steps shown are explained. Figure 4 The entity responsible for executing the steps is Figure 1 400 terminal devices.

[0101] In step 401, based on the first face image, an image processing model is invoked to perform feature extraction processing to obtain the first image features.

[0102] For example, the requirements for face quality are deduced from face recognition, namely, the need for clear facial features. Data sets and algorithms are constructed based on this idea. To better serve face recognition, the clarity of facial features can be used to judge face quality. Taking a welcoming scenario as an example, since welcoming guests requires high real-time performance, the features extracted from the deep semantic layer channels are related to facial features. Therefore, a channel attention mechanism can be added to the deep features for recognition. This application uses state-of-the-art (SOTA) lightweight networks, such as the Reparameterized Visual Geometry Group Network (RepVGG), among which the RepVGGA0 model is the network that best balances accuracy and speed. An image processing model is built based on the RepVGGA0 model. The advantage of the RepVGGA0 model is that it can significantly accelerate 3x3 convolutional units and improve accuracy using reparameterization techniques.

[0103] The image processing model is explained below with reference to the accompanying diagram. Figure 5A , Figure 5A This is a first structural diagram of the image processing model provided in this application embodiment; the RepVGGA0 model has a structure similar to the Residual Network 501 (ResNet), and is used to extract features from the input image at multiple levels. The RepVGG training phase 502 refers to the RepVGGA0 model during the training phase, and the RepVGG inference phase 503 refers to the RepVGGA0 model during the application phase. The RepVGGA0 model contains multiple convolutional layers and multiple Rectified Linear Unit (ReLU) layers.

[0104] refer to Figure 5B , Figure 5B This is a schematic diagram of the second structure of the image processing model provided in the embodiments of this application; Figure 5B Structure 504 is a component of the RepVGGA0 model. The RepVGGA0 model consists of multiple structures 504, including: a linear rectified function 5041, convolutional layers 5043 and 5042, a batch normalization layer 5044, an additive layer 5046, and an additive layer 5045. Convolutional layers 5043 and 5042 have different sizes. The features output from the linear rectified function 5041 are input into convolutional layers 5043, 5042, and 5044 for processing. The processing results from each layer are summed to obtain the output features.

[0105] In some embodiments, the inference resolution is set to 112. Based on the RepVGGA0-112 classification network, considering the correlation between the features extracted from the deep semantic layer channels and facial features, a channel attention mechanism is introduced into the deep features for recognition. This mechanism is the Squeeze-and-Excitation Network (SENet). The channel attention mechanism module highlights the role of core channels by modeling channel relationships. (Refer to...) Figure 5C , Figure 5C This is a third structural schematic diagram of the image processing model provided in the embodiments of this application, used to characterize the principle of the channel attention mechanism; wherein, It is the basic feature map (Transform Feature). Yes Obtained by performing a squeeze operation. Yes Obtained through activation. The result is obtained through a channel attention mechanism. Since the core deep semantic channels generally reflect facial details, highlighting the learning of facial feature details can obviously help with facial gender recognition tasks. Compared to adding an attention mechanism to each stage, adding an attention mechanism only to the deep semantic layer can better help with recognition.

[0106] In some embodiments, networks such as GhostNet, Pelee, PaddlePaddle Lightweight Convolutional Network (PPLCNET), MobileOne, and MobileFace can also be used.

[0107] In step 402, based on the first image features, an image processing model is invoked to perform prediction processing to obtain the predicted probabilities of the first face image for different resolution types.

[0108] For example, the output of the RepVGGA0 model in the application phase is the predicted probability that the input face image belongs to different sharpness types. Sharpness types include: blurry, first sharpness, and second sharpness. The sharpness of the first sharpness type is less than that of the second sharpness type, and the first sharpness type is greater than that of the blurry type. Prediction processing can be achieved as follows: Keypoint extraction is performed based on the first image features to obtain facial keypoints. Based on these keypoints, the keypoints of each facial feature are concatenated according to their sequence number. A geometric fitting algorithm connects the discrete points into continuous, smooth closed boundaries, and these boundaries must conform to the natural shape of the facial features (e.g., eyes are elliptical, eyebrows are arc-shaped, and the mouth is a closed curve). The area enclosed by the closed boundaries is taken as the facial feature region, and the pixels with coordinates within this region are taken as the pixels of the facial feature region. The facial feature region is 1 (white, highlight), and the background is 0 (black, low brightness), i.e., a binary heatmap. (Reference) Figure 7A , Figure 7A This is a schematic diagram of facial key points provided in an embodiment of this application; see reference. Figure 7B , Figure 7B This is a schematic diagram of the facial feature heatmap provided in this embodiment. The sample face image 701 is multiplied by the facial feature heatmap 702. Since the parameter in the facial feature heatmap is 0, the result does not contain black areas, but only the face sample image corresponding to the white areas of the facial feature heatmap 702, i.e., the facial features. Based on the facial feature image, the image processing model described above is used to predict the prediction probability of the first face image for different resolution types.

[0109] In step 403, a linear operation is performed based on the predicted probability corresponding to the sharpness type to obtain the sharpness index of the first face image.

[0110] For example, the categories are divided into three classes. When processing sharpness, the idea of ​​Digital Enhancement for Extreme Clarity (DEX) is used. Assuming that y = {0, 60, 100} is assigned, that is, the preset value corresponding to blurriness is 0 points, the preset value corresponding to relatively sharpness is 60 points, and the preset value corresponding to sharpness is 100 points. The result obtained by the neural network is the probability o = {o1, o2, o3} of these three classes, which is used to calculate the final sharpness index E of this image. The formula for calculating the sharpness index E is (1): (1) The fuzzy judgment made under this approach yields a quantifiable result. Assume the input image processing model has a clear face image, with the sharpness index E hovering around 100. If the face image is blurry, the sharpness index E hovers around 0. If the face image is relatively clear, the sharpness index E hovers around 60. If the blurriness of the face image is more or less clear, the sharpness index E score will be less than or equal to 60; if the blurriness of the face image is more or less clear, the sharpness index E score will be greater than or equal to 60. Therefore, using 60 as a preset threshold for fuzziness can help improve the accuracy of face recognition.

[0111] In step 404, the first face image is filtered according to the clarity index, and the selected first face image is used for face recognition.

[0112] For example, based on the clarity index of the first face image, it is determined whether to retain the first face image. If the clarity index of the first face image is greater than a preset threshold, face recognition is performed based on the first face image; otherwise, if the clarity index of the first face image is less than the preset threshold, the first face image is discarded.

[0113] In some embodiments, before step 401, the image processing model is trained by: constructing a first auxiliary branch for the image processing model, which is used to perform multi-level feature extraction on sample face images to obtain predicted facial feature images; obtaining a first loss between the predicted facial feature images and the actual extracted facial feature images; the extracted facial feature images are obtained by multiplying the facial feature heatmap with the sample face image matrix; constructing a second auxiliary branch for the image processing model, which is used to perform multi-level feature extraction on sample face images of different resolutions to obtain predicted facial feature images of different resolutions; obtaining a second loss between the predicted facial feature images and the actual extracted facial feature images. The image processing model is trained based on the losses obtained from the different auxiliary branches. The network structure of the second auxiliary branch is exactly the same as that of the first auxiliary branch, but the parameters are different. The auxiliary branches only participate in training; in actual applications, only [the network structure of the second auxiliary branch is different from that of the first auxiliary branch]. Figure 5A The image processing model participates in inference. In summary, the first auxiliary branch helps the network extract information related to facial features, and the second auxiliary branch further helps the network extract features indicating whether facial features are clear or not.

[0114] refer to Figure 6A , Figure 6AThis is a schematic diagram of the fourth structure of the image processing model provided in this application embodiment. The structure of the first auxiliary branch references the U-shaped network (UNet) structure, using a 3x3 auxiliary convolutional layer (conv_aux) to extract features, concatenating them with the features from the previous level, and then adjusting the dimensions through another 3x3 convolutional layer (conv_aux). This process is repeated to obtain the final predicted facial feature image (the first predicted image mentioned above). A first loss is used to establish the loss between the predicted facial feature image and the actual extracted facial feature image. The model is trained based on this loss, which allows the features of the model's inference process to reconstruct the facial feature image, thereby helping the model better understand the facial features from the image. The first loss can be the mean squared error loss. It is expressed as the following formula (2): (2) Where P represents the pixel value of the predicted image, T represents the pixel value of the extracted facial features image, and H×W represents the image size. The first loss obtained through the first auxiliary branch completes the reconstruction of the facial features. If further fine-grained classification of the reconstructed facial features is needed, a second auxiliary branch (aux-guide 2) can be introduced; see reference... Figure 6B , Figure 6B This is a fifth structural schematic diagram of the image processing model provided in this application embodiment; to distinguish between "relatively clear" and "clear," the relatively clear heatmap is blurred and noise-added to reflect that only blurred facial features are visible in the relatively clear image. (Reference) Figure 7C , Figure 7C This is a schematic diagram of facial feature heatmaps with different resolutions provided in an embodiment of this application. For a clear image, the second auxiliary branch is used to predict a clear facial feature heatmap; for a relatively clear image, the second auxiliary branch is used to predict a relatively clear facial feature heatmap with added noise; and for a blurry image, a blurry heatmap cannot be predicted. The second loss can also be the mean squared error loss. The sum of the first loss and the second loss is used as the total loss. Based on the total loss, backpropagation is used to update the parameters of the image processing model to obtain the trained model.

[0115] In this embodiment, the face quality is judged in real time by combining the clarity of facial features. The network structure of the image processing model is designed by working backward from the requirements of facial recognition for the clarity of facial features. This allows for better selection from the facial trajectory, enabling face recognition to achieve better results in welcoming application scenarios.

[0116] The following description continues to illustrate the exemplary structure of the face image processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules stored in the face image processing device 455 in the memory 450 may include: a loss acquisition module 4551, used to call a first auxiliary module to perform feature extraction based on a first sample face image to obtain a first predicted image; determine a first loss based on the difference between the first predicted image and the first facial feature extraction image; call a second auxiliary module to perform feature extraction based on second sample face images of different resolution types to obtain second predicted images of different resolution types; determine a second loss based on the difference between the second predicted images of different resolution types and the corresponding second facial feature extraction images; a training module 4552, used to update the parameters of the image processing model based on the first loss and the second loss to obtain a trained image processing model; an image processing module 4553, used to call the trained image processing model to perform feature extraction based on a video frame image to obtain a first image feature; the image processing module 4553 is also used to call the trained image processing model to predict the resolution index of the video frame image based on the first image feature.

[0117] In some embodiments, the image processing module 4553 is configured to call the trained image processing model based on the first image features to perform probability prediction processing, obtain a first predicted probability that the video frame image belongs to different sharpness types; obtain a preset value corresponding to each sharpness type; and perform a linear operation based on the first predicted probability of each sharpness type and the preset value to obtain the sharpness index of the video frame image.

[0118] In some embodiments, the image processing module 4553 is configured to multiply the first predicted probability by the preset value for each of the resolution types to obtain a first product; and to use the sum of each first product as the resolution index of the video frame image.

[0119] In some embodiments, the loss acquisition module 4551 is configured to, before determining the first loss based on the difference between the first predicted image and the first facial feature extraction image, perform facial feature segmentation on the first sample face image to obtain the first facial feature extraction image by: performing key point recognition on the first sample face image to obtain key points corresponding to the facial features in the first sample face image; connecting the key points of each part of the facial features to obtain a first contour line of each part; performing curve processing on each first contour line to obtain a second contour line; taking the area surrounded by the second contour line in the first sample face image as the facial feature region; constructing a facial feature heatmap based on the facial feature region, wherein the pixel value outside the facial feature region in the facial feature heatmap is 0; performing matrix multiplication on the first sample face image and the facial feature heatmap to obtain the first facial feature extraction image.

[0120] In some embodiments, the loss acquisition module 4551 is configured to determine the second pixel corresponding to each first pixel in the first predicted image in the first facial feature extraction image; determine a first difference between the first pixel value of each first pixel and the second pixel value of the corresponding second pixel; determine the square of each first difference; and take the average of each square as the first loss.

[0121] In some embodiments, the loss acquisition module 4551 is used to perform noise processing on the first sample face image to different degrees before calling the second auxiliary module to perform feature extraction based on the second sample face image of different resolution types to obtain the second prediction image of different resolution types; before determining the second loss based on the difference between the second prediction image of different resolution types and the corresponding second facial feature extraction image, the method further includes: performing facial feature segmentation on the second sample face image of different resolution types to obtain the corresponding second facial feature extraction image.

[0122] In some embodiments, the image processing module 4553 is used to extract and process the video frame image to obtain a second image feature; to compress the second image feature to obtain a third image feature with a lower dimension than the second image feature; and to activate the third image feature to obtain the first image feature.

[0123] In some embodiments, the image processing module 4553 is configured to, after predicting the sharpness index of the video frame image by calling the trained image processing model based on the first image features, discard the video frame image when the sharpness index of the video frame image is less than a preset threshold; and perform identity recognition on the face in the video frame image when the sharpness index of the video frame image is greater than or equal to the preset threshold.

[0124] This application provides a computer program product, which includes a computer program or computer-executable instructions. A processor executes the computer program or computer-executable instructions to implement the face image processing method described in this application.

[0125] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the face image processing method provided in this application. For example, ... Figure 3A The method for processing human face images is shown.

[0126] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0127] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0128] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0129] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0130] In summary, by constructing a first auxiliary module and a second auxiliary module in addition to the image processing model, the embodiments of this application can obtain the loss between the first predicted image and the first facial feature extraction image, thereby improving the sensitivity of the image processing model to the face region. The image processing model can determine the facial feature details in the face image and then judge the sharpness based on the features of the facial feature details. It can also obtain the loss between the second predicted image of different sharpness types and the corresponding second facial feature extraction image, thereby improving the sensitivity of the image processing model to different sharpness of the face region. The image processing model can determine the facial feature details in face images of different sharpness and then judge the sharpness based on the features of the facial feature details. In the application stage, the first auxiliary module and the second auxiliary module do not participate in the calculation process of the image processing model, saving the storage space occupied by the image processing model and improving the computational efficiency and the accuracy of sharpness prediction.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A face image processing method, characterized in that, The method includes: Based on the first sample face image, the first auxiliary module is invoked to perform feature extraction to obtain the first predicted image; The first loss is determined based on the difference between the first predicted image and the first facial feature extraction image; Based on the second sample face images of different resolution types, the second auxiliary module is called to extract features and obtain second predicted images of different resolution types; The second loss is determined based on the difference between the second predicted image of different resolution types and the corresponding second facial feature extraction image. The parameters of the image processing model are updated based on the first loss and the second loss to obtain the trained image processing model. Based on the video frame images, the trained image processing model is invoked to extract features and obtain the first image features; Based on the first image features, the trained image processing model is invoked to predict the sharpness index of the video frame image.

2. The method according to claim 1, characterized in that, The step of predicting the sharpness index of the video frame image by calling the trained image processing model based on the first image features includes: Based on the first image features, the trained image processing model is invoked to perform probability prediction processing to obtain the first predicted probability that the video frame image belongs to different resolution types. Obtain the preset value corresponding to each of the aforementioned resolution types; The clarity index of the video frame image is obtained by performing a linear operation on the first predicted probability of each clarity type and the preset value.

3. The method according to claim 2, characterized in that, The process of performing a linear operation between the first predicted probability for each of the aforementioned sharpness types and the preset value to obtain the sharpness index of the video frame image includes: For each of the aforementioned clarity types, the first predicted probability is multiplied by the preset value to obtain a first product; The sum of each first product is used as an indicator of the sharpness of the video frame image.

4. The method according to claim 1, characterized in that, Before determining the first loss based on the difference between the first predicted image and the first facial feature extraction image, the method further includes: The first sample face image is segmented into facial features using the following method to obtain the first facial feature extraction image: Key point recognition is performed on the first sample face image to obtain the key points corresponding to the facial features in the first sample face image; Connect the key points of each part of the facial features to obtain the first contour line of each part; Each of the first contour lines is processed to obtain the second contour line; The area surrounded by the second contour line in the first sample face image is taken as the facial feature area; A heat map of facial features is constructed based on the facial feature regions, wherein the pixel value outside the facial feature regions in the heat map of facial features is 0; The first sample face image is multiplied by the facial feature heatmap to obtain the first facial feature extraction image.

5. The method according to claim 1, characterized in that, The determination of the first loss based on the difference between the first predicted image and the first facial feature extraction image includes: Determine the second pixel corresponding to each first pixel in the first predicted image in the first facial feature extraction image; Determine the first difference between the first pixel value of each first pixel point and the second pixel value of the corresponding second pixel point; Determine the square of each of the first differences; The average of each squared value is taken as the first loss.

6. The method according to claim 1, characterized in that, Before calling the second auxiliary module to perform feature extraction based on the second sample face images of different resolution types to obtain the second predicted images of different resolution types, the method further includes: The first sample face image is subjected to noise processing of different degrees to obtain second sample face images of different clarity types; Before determining the second loss based on the difference between the second predicted image and the corresponding second facial feature extraction image based on different resolution types, the method further includes: Facial features were segmented for second sample face images of different resolution types to obtain corresponding second facial feature extraction images.

7. The method according to claim 1, characterized in that, The step of extracting features from the trained image processing model based on video frame images to obtain the first image features includes: The video frame images are extracted and processed to obtain the second image features; The second image features are compressed to obtain a third image feature with a lower dimension than the second image features; The third image feature is activated to obtain the first image feature.

8. The method according to claim 1, characterized in that, After predicting the sharpness index of the video frame image based on the first image feature by calling the trained image processing model, the method further includes: When the clarity index of the video frame image is less than a preset threshold, the video frame image is discarded; When the clarity index of the video frame image is greater than or equal to a preset threshold, the face in the video frame image is identified.

9. A face image processing device, characterized in that, The device includes: The loss acquisition module is used to call the first auxiliary module to perform feature extraction based on the first sample face image to obtain the first predicted image; determine the first loss based on the difference between the first predicted image and the first facial feature extraction image; call the second auxiliary module to perform feature extraction based on the second sample face images of different resolution types to obtain the second predicted images of different resolution types; determine the second loss based on the difference between the second predicted images of different resolution types and the corresponding second facial feature extraction images. The training module is used to update the parameters of the image processing model based on the first loss and the second loss to obtain the trained image processing model. The image processing module is used to extract features based on the trained image processing model using video frame images to obtain the first image features; The image processing module is further configured to, based on the first image features, invoke the trained image processing model to predict the sharpness index of the video frame image.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the face image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the face image processing method according to any one of claims 1 to 8 is implemented.

Images