Image library construction method, electronic device, and storage medium

Abstract

The application relates to the field of image processing, and provides an image library construction method, an electronic device and a storage medium. The method performs face detection on an obtained input image, determines a detection position of a face detection frame in the input image, intercepts a face image from the input image based on the detection position, performs face key point detection on the face image, obtains multiple face key point coordinates, generates multiple target key point coordinates corresponding to the multiple face key point coordinates according to a target position of the face image and the multiple face key point coordinates, performs alignment processing on the input image according to the multiple target key point coordinates and preset standard face coordinates, obtains an aligned image, and establishes a target image library according to a face discrimination result of the aligned image. Through the construction of the target image library, the input image does not need to be repeatedly preprocessed when a face-related task is processed, and the task execution efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image library construction method, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence, face-related tasks such as face recognition, face detection, and face stylization based on deep learning have emerged. To implement these face-related tasks, it is typically necessary to collect and process face data. However, due to differences in the format of the collected face data, the lighting conditions during shooting, and the angle between the face and the camera, the collected face images may not meet the requirements of the task. Therefore, each time a face-related task is processed, it is usually necessary to repeatedly preprocess the collected face images. Furthermore, because the collected face images occupy a large amount of storage space, the time spent processing tasks based on these images is excessive, resulting in low task execution efficiency. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image library construction method, electronic device and storage medium that can solve the technical problem of low task execution efficiency caused by repeated image preprocessing in each face-related task.

[0004] On one hand, this application proposes an image library construction method, which includes: performing face detection on an acquired input image to determine the detection position of a face detection box in the input image; extracting a face image from the input image based on the detection position; performing facial key point detection on the face image to obtain multiple facial key point coordinates; generating multiple target key point coordinates corresponding to the multiple facial key point coordinates according to the target position of the face image and the multiple facial key point coordinates; performing alignment processing on the input image according to the multiple target key point coordinates and preset standard face coordinates to obtain an aligned image; and establishing a target image library based on the face discrimination result of the aligned image.

[0005] According to a preferred embodiment of this application, the step of performing face detection on the acquired input image and determining the detection location of the face detection box in the input image includes: acquiring a pre-trained face detection model, the face detection model including a feature extraction network and a detection box generation network, the feature extraction network including a fine feature extraction layer and a coarse feature extraction layer, the detection box generation network including a face recognition layer, a target convolutional layer, a target fusion layer, the fine feature extraction layer, and a detection box localization layer; extracting target coarse features of the input image based on the feature extraction network; identifying primary face features from the target coarse features based on the face recognition layer; generating aggregate features of the primary face features using the target convolutional layer and the target fusion layer; extracting target precision features of the aggregate features based on the fine feature extraction layer; identifying target face features from the target precision features based on the face recognition layer; locating the target face features based on the detection box localization layer to obtain the face detection box; and using the position of the face detection box in the input image as the detection location.

[0006] According to a preferred embodiment of this application, the step of extracting a face image from the input image based on the detection location includes: identifying the initial height and initial width of the face detection box based on the detection location; processing the initial height based on a preset ratio to obtain a target height, and processing the initial width based on the preset ratio to obtain a target width; calculating the height difference between the target height and the initial height, and calculating the width difference between the target width and the target width; identifying the target location based on the height difference, the width difference, and the detection location; and extracting the region corresponding to the target location from the input image as the face image.

[0007] According to a preferred embodiment of this application, the detection position includes multiple initial coordinate points, the target position includes multiple target points, and the step of identifying the target position based on the height difference, the width difference, and the detection position includes: determining the initial coordinate point with the minimum horizontal coordinate value and the minimum vertical coordinate value as the first coordinate point, and determining the initial coordinate point with the maximum horizontal coordinate value and the maximum vertical coordinate value as the second coordinate point, wherein the first coordinate point includes a first horizontal coordinate value and a first vertical coordinate value, and the second coordinate point includes a second horizontal coordinate value and a second vertical coordinate value; calculating the width ratio of the width difference to a preset value, and calculating the height ratio of the height difference to the preset value; generating a target point corresponding to the first coordinate point based on the difference between the first horizontal coordinate value and the width ratio, and the difference between the first vertical coordinate value and the height ratio, and generating a target point corresponding to the second coordinate point based on the sum of the second horizontal coordinate value and the width ratio, and the sum of the second vertical coordinate value and the height ratio.

[0008] According to a preferred embodiment of this application, the step of detecting facial key points in the face image to obtain multiple facial key point coordinates includes: acquiring a pre-trained key point detection model, the key point detection model including a feature detection network, a key point detection network, a key point adjustment network, and a key point transformation network; extracting key point region features from the face image based on the feature detection network; detecting initial key point features from the key point region features based on the key point detection network; adjusting the initial key point features based on the key point adjustment network to obtain target key point features; and transforming the target key point features based on the key point transformation network to obtain the multiple facial key point coordinates.

[0009] According to a preferred embodiment of this application, the target location includes multiple location point coordinates. The step of generating multiple target key point coordinates corresponding to the multiple facial key point coordinates based on the target location of the face image and the multiple facial key point coordinates includes: determining the location point coordinates with the smallest horizontal coordinate value and the smallest vertical coordinate value as the target location point coordinates; calculating the sum of each facial key point coordinate and the target location point coordinates to obtain the multiple target key point coordinates.

[0010] According to a preferred embodiment of this application, the step of aligning the input image based on the coordinates of the plurality of target key points and the preset standard face coordinates to obtain an aligned image includes: calculating a similarity transformation matrix between the input image and the standard face image corresponding to the preset standard face coordinates based on the coordinates of the plurality of target key points and the preset standard face coordinates; transforming the initial coordinates of each pixel in the input image based on the similarity transformation matrix to obtain the transformed coordinates corresponding to each pixel in the input image; and generating the aligned image based on the pixels in the input image corresponding to the transformed coordinates.

[0011] According to a preferred embodiment of this application, establishing a target image library based on the face discrimination result of the aligned image includes: constructing a face discrimination model based on preset image requirements; judging the aligned image based on the face discrimination model to obtain the face discrimination result; if the face discrimination result is greater than a preset confidence level, storing the mapping information between the aligned image and the coordinates of the plurality of target key points in the target image library.

[0012] On the other hand, this application also proposes an image library construction apparatus, which includes: a detection unit for performing face detection on an acquired input image and determining the detection position of a face detection box in the input image; a cropping unit for cropping a face image from the input image based on the detection position; the detection unit is further configured to perform facial key point detection on the face image to obtain multiple facial key point coordinates; a generation unit for generating multiple target key point coordinates corresponding to the multiple facial key point coordinates based on the target position of the face image and the multiple facial key point coordinates; an alignment unit for aligning the input image based on the multiple target key point coordinates and a preset standard face coordinate to obtain an aligned image; and a building unit for building a target image library based on the face discrimination result of the aligned image.

[0013] On the other hand, this application also proposes an electronic device, the electronic device comprising: a memory for storing computer-readable instructions; and a processor for executing the computer-readable instructions stored in the memory to implement the image library construction method.

[0014] On the other hand, this application also proposes a computer-readable storage medium storing computer-readable instructions, which are executed by a processor in an electronic device to implement the image library construction method.

[0015] As can be seen from the above technical solutions, this application extracts face images from input images by detecting positions. This not only preserves face information completely but also reduces the space occupied by the face image, while improving the efficiency of key point detection in the face image, thereby improving image preprocessing efficiency. Furthermore, by combining the coordinates of multiple target key points and preset standard face coordinates, the input image is aligned, making the aligned image closer to the standard face image. This avoids the loss of image information caused by different shooting lighting methods and camera angles, improving the robustness of the aligned image. Moreover, by judging faces in the aligned image, the input image can be filtered to remove images that do not meet the face requirements, thereby improving the accuracy of the target image library. In addition, by establishing a target image library, since it is not necessary to repeatedly preprocess the acquired input images each time face-related tasks are processed, the task execution efficiency is improved. Attached Figure Description

[0016] Figure 1 This is an application environment diagram of the image library construction method in the embodiments of this application.

[0017] Figure 2 This is a flowchart of the image library construction method in the embodiments of this application.

[0018] Figure 3 This is a network structure diagram of the coarse feature extraction layer in the embodiments of this application.

[0019] Figure 4 This is a network structure diagram of the fine feature extraction layer in the embodiments of this application.

[0020] Figure 5 This is a network structure diagram of the hierarchical identification layer in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of key points of a standard face image in an embodiment of this application.

[0022] Figure 7 This is a network structure diagram of the feature detection network in the embodiments of this application.

[0023] Figure 8 This is a network structure diagram of the key point detection network in the embodiments of this application.

[0024] Figure 9 This is a network structure diagram of the key point adjustment network in the embodiments of this application.

[0025] Figure 10 This is a network structure diagram of the key point conversion network in the embodiments of this application.

[0026] Figure 11 This is a functional block diagram of the image library construction device in the embodiments of this application.

[0027] Figure 12 This is a schematic diagram of the structure of an electronic device that implements the image library construction method in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0030] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0031] Face tracking methods can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0032] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0033] like Figure 1 The diagram shown is an application environment diagram of the image library construction method in the embodiments of this application.

[0034] The image library construction method is applied in electronic device 1, which communicates with imaging device 2. Imaging device 2 is used to capture various types of images, such as images of faces. Electronic device 1 is used to preprocess the images captured by imaging device 2.

[0035] In this embodiment of the application, the image library construction method is applied to one or more electronic devices 1. The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0036] Electronic device 1 can be any electronic product that can interact with a user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0037] Electronic device 1 may include network devices and / or user devices. Among them, network devices include, but are not limited to, a single network electronic device, a group of electronic devices consisting of multiple network electronic devices, or a cloud based on cloud computing consisting of a large number of hosts or network electronic devices.

[0038] The network in which electronic device 1 is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0039] The shooting device 2 can be a photographic device independent of the electronic device 1, or it can be the camera of the electronic device 1. When the shooting device 2 is a photographic device independent of the electronic device 1, the shooting device 2 communicates with the electronic device 1. The shooting device 2 can be installed at the entrance of a residential area, a bank, or other similar locations.

[0040] like Figure 2 The diagram shown is a flowchart of an image library construction method in an embodiment of this application. The image library construction method is applied to electronic devices (e.g.,...) Figure 1 In the electronic device 1), the order of steps in the flowchart can be changed according to different needs, and some steps can be omitted.

[0041] 101. Perform face detection on the acquired input image to determine the detection location of the face detection box in the input image.

[0042] In at least one embodiment of this application, the input image can be an image captured by the imaging device 2 under any lighting conditions, or an image directly transmitted by the user to the electronic device 1 via the input device. The input image is not limited to images of faces, and its size can be a preset size, such as 160mm * 96mm. The face detection box can be a detection box that detects faces from the input image using a pre-trained face detection model. The detection location refers to the specific position of the face detection box in the input image, which includes multiple initial coordinate points of the face detection box in the input image. These multiple initial coordinate points can be determined based on a coordinate system constructed from the input image. Specifically, the coordinate system can be constructed using the wide edge of the input image as the x-axis and the high edge of the input image as the y-axis.

[0043] In at least one embodiment of this application, the electronic device performs face detection on the acquired input image and determines the detection location of the face detection box in the input image, including: the electronic device acquires a pre-trained face detection model, the face detection model including a feature extraction network and a detection box generation network, the feature extraction network including a fine feature extraction layer and a coarse feature extraction layer, the detection box generation network including a face recognition layer, a target convolutional layer, a target fusion layer, a fine feature extraction layer and a detection box localization layer, the electronic device extracts the target coarse features of the input image based on the feature extraction network, and identifies primary face features from the target coarse features based on the face recognition layer, further, the electronic device uses the target convolutional layer and the target fusion layer to generate aggregated features of the primary face features, and extracts the target precision features of the aggregated features based on the fine feature extraction layer, and further still, the electronic device identifies the target face features from the target precision features based on the face recognition layer, and locates the target face features based on the detection box localization layer to obtain a face detection box, and uses the position of the face detection box in the input image as the detection location.

[0044] The coarse feature extraction layer includes a standard convolutional layer, a depthwise separable convolutional layer, an activation layer, and a fusion layer, such as... Figure 3 As shown, the coarse feature extraction layer can include 3 standard convolutional layers, 2 depthwise separable convolutional layers, 2 activation layers, and 1 fusion layer. Figure 3 The specific connection structure of each network layer in the coarse feature extraction layer is also shown: the standard convolutional layer is connected in series with two depthwise separable convolutional layers, each depthwise separable convolutional layer is connected in series with the corresponding activation layer, each activation layer is connected in series with the corresponding standard convolutional layer, and each standard convolutional layer is connected in series with the fusion layer.

[0045] The fine feature extraction layer includes a standard convolutional layer, a depthwise separable convolutional layer, an activation layer, a coarse feature extraction layer, and a stacking layer, such as... Figure 4 As shown, the fine feature extraction layer can include two standard convolutional layers, one depthwise separable convolutional layer, two activation layers, two coarse feature extraction layers, and one stacking layer. Figure 4 It also demonstrates the specific connection structure of each network layer in the fine feature extraction layer: standard convolutional layer connected in series with activation layer, activation layer connected in series with depthwise separable convolutional layer, depthwise separable convolutional layer connected in series with standard convolutional layer, standard convolutional layer connected in series with activation layer, activation layer connected in series with coarse feature extraction layer, coarse feature extraction layer connected in series with another coarse feature extraction layer, and both coarse feature extraction layers are connected in series with stacked layers.

[0046] like Figure 5As shown, the face recognition layer can include two hierarchical recognition layers. Each hierarchical recognition layer includes three standard convolutional layers, two depthwise separable convolutional layers, two activation layers, one pooling layer, one normalization layer, one fusion layer, and one stacking layer. Figure 5 The specific connection structure of each network layer in the face recognition layer is also shown. The standard convolutional layer is connected in series with the depthwise separable convolutional and pooling layers, the depthwise separable convolutional layer is connected in series with the activation layer, the activation layer is connected in series with the standard convolutional layer (A layer), the pooling layer is connected in series with the depthwise separable convolutional and pooling layers, the depthwise separable convolutional layer is connected in series with the activation layer, the activation layer is connected in series with the standard convolutional layer, the standard convolutional layer is connected in series with the normalization layer, the A layer and the normalization layer are both connected in series with the fusion layer, and the fusion layer is connected in series with the overlay layer.

[0047] The detection box localization layer includes multiple transformation layers and fusion layers. Each transformation layer consists of a cascaded structure of a standard convolutional layer and a reshape() function layer.

[0048] By extracting features from the input image using a feature extraction network, the size of the input image can be compressed while preserving its features, thus improving the efficiency of face detection bounding box generation. Furthermore, since the face detection bounding box is not a fixed-size bounding box, but rather the target face feature is located using a bounding box localization layer, the accuracy of face detection box localization is improved, thereby increasing the accuracy of detection location generation.

[0049] Specifically, the electronic device extracts target coarse features of the input image based on the feature extraction network, including: the electronic device extracts primary precision features of the input image based on the fine feature extraction layer, and extracts primary coarse features of the precision features based on the coarse feature extraction layer; further, the electronic device extracts target coarse features of the primary coarse features based on the fine feature extraction layer.

[0050] By performing feature extraction multiple times on the input image, the accuracy of generating target coarse features can be improved.

[0051] 102. Based on the detection location, extract the face image from the input image.

[0052] In at least one embodiment of this application, a face image refers to an image containing facial information in the input image, and the size of the face image is smaller than the size of the input image. The size of the face image can be a preset size, such as 48mm*48mm.

[0053] In at least one embodiment of this application, the electronic device extracts a face image from an input image based on a detection location, comprising: the electronic device identifying the initial height and initial width of a face detection box based on the detection location, processing the initial height according to a preset ratio to obtain a target height, processing the initial width according to a preset ratio to obtain a target width, the electronic device calculating the height difference between the target height and the initial height, and calculating the width difference between the target width and the target width, identifying the target location based on the height difference, the width difference, and the detection location, and extracting the area corresponding to the target location from the input image as a face image.

[0054] In order to ensure that the face image contains all the face information in the input image, the preset ratio is usually set to a value greater than 1.

[0055] By processing the initial height and width according to a preset ratio, the size of the face detection box can be increased, thereby allowing the enlarged face detection box to contain more face information and effectively avoiding the loss of face information in the face image.

[0056] Specifically, the detection location includes multiple initial coordinate points, and the target location includes multiple target points. The electronic device identifies the target location based on the height difference, width difference, and detection location by: determining the initial coordinate point with the minimum horizontal coordinate value and the minimum vertical coordinate value as the first coordinate point, and determining the initial coordinate point with the maximum horizontal coordinate value and the maximum vertical coordinate value as the second coordinate point. The first coordinate point includes a first horizontal coordinate value and a first vertical coordinate value, and the second coordinate point includes a second horizontal coordinate value and a second vertical coordinate value. The electronic device calculates the width ratio of the width difference to a preset value, and calculates the height ratio of the height difference to a preset value. Based on the difference between the first horizontal coordinate value and the width ratio, and the difference between the first vertical coordinate value and the height ratio, the electronic device generates a target point corresponding to the first coordinate point. Based on the sum of the second horizontal coordinate value and the width ratio, and the sum of the second vertical coordinate value and the height ratio, the electronic device generates a target point corresponding to the second coordinate point.

[0057] The preset value can be set according to actual needs; for example, the preset value can be set to 2.

[0058] By combining preset values ​​to determine the target location point, the target location point can be reasonably determined, thereby reducing the loss of facial information in the face image to a certain extent.

[0059] 103. Facial landmark detection is performed on the face image to obtain the coordinates of multiple facial landmarks.

[0060] In at least one embodiment of this application, multiple facial keypoint coordinates refer to multiple coordinate points identified from a facial image based on a pre-trained keypoint detection model, and each facial keypoint coordinate corresponds to a preset standard facial coordinate. The preset standard facial coordinates refer to the coordinates corresponding to the keypoints of a standard facial image, such as... Figure 6 As shown, Figure 6 The key features of a standard human face image are presented in the image.

[0061] In at least one embodiment of this application, the electronic device performs facial landmark detection on a face image to obtain multiple facial landmark coordinates, including: the electronic device acquiring a pre-trained landmark detection model, the landmark detection model including a feature detection network, a landmark detection network, a landmark adjustment network, and a landmark transformation network; the electronic device extracting landmark region features from the face image based on the feature detection network, detecting initial landmark features from the landmark region features based on the landmark detection network, further adjusting the initial landmark features based on the landmark adjustment network to obtain target landmark features, and further transforming the target landmark features based on the landmark transformation network to obtain multiple facial landmark coordinates.

[0062] Among them, such as Figure 7 As shown, the feature detection network can include one standard convolutional layer, one activation layer, and two primary feature extraction layers. Each primary feature extraction layer can include one depthwise separable convolutional layer, one standard convolutional layer, one stacking layer, and one activation layer. Figure 7 It also demonstrates the connection structure of each network layer in the feature detection network: standard convolutional layers are connected in series with activation layers; activation layers are connected in series with stacked layers (B layers) and depthwise separable convolutional layers; depthwise separable convolutional layers are connected in series with another standard convolutional layer; another standard convolutional layer is connected in series with B layers; B layers are connected in series with activation layers; and activation layers are connected in series with primary feature extraction layers.

[0063] like Figure 8 As shown, the keypoint detection network can include one depthwise separable convolutional layer, one feature detection network, one primary feature extraction layer, and one pooling layer. Figure 8 It also demonstrates the connection structure of each network layer in the keypoint detection network, with depthwise separable convolutional layers and pooling layers connected in series with the feature detection network, and the feature detection network connected in series with the primary feature extraction layer.

[0064] like Figure 9 As shown, the keypoint adjustment network can include one keypoint detection network and three primary feature extraction layers. Figure 9It also demonstrates the connection structure of each network layer in the key point adjustment network, where the key point detection network is connected in series with the first primary feature extraction layer, the first primary feature extraction layer is connected in series with the second primary feature extraction layer, and the second primary feature extraction layer is connected in series with the third primary feature extraction layer.

[0065] like Figure 10 As shown, the keypoint transformation network can include two standard convolutional layers, one activation layer, and one reshape() function layer. Figure 10 It also demonstrates the connection structure of each network layer in the keypoint transformation network: a standard convolutional layer is connected in series with an activation layer, an activation layer is connected in series with another standard convolutional layer, and another standard convolutional layer is connected in series with a reshape() function layer.

[0066] In other implementations, the number of network layers can vary when training the keypoint detection model in an electronic device. For example, the keypoint transformation network may include five standard convolutional layers.

[0067] By further adjusting the detected initial keypoints through a keypoint adjustment network, the accuracy of generating target keypoint features can be improved, thereby improving the accuracy of generating coordinates of multiple facial keypoints.

[0068] 104. Based on the target location of the face image and the coordinates of multiple facial key points, generate multiple target key point coordinates corresponding to the multiple facial key point coordinates.

[0069] In at least one embodiment of this application, the target location includes the coordinates of multiple location points in the face image. The multiple target key point coordinates refer to the coordinate information of the key points of the face image corresponding to the key points in the input image.

[0070] In at least one embodiment of this application, the electronic device generates multiple target key point coordinates corresponding to the multiple face key point coordinates based on the target location of the face image and the coordinates of multiple face key points. This includes: the electronic device determines the coordinates of the location point with the smallest horizontal coordinate value and the smallest vertical coordinate value as the target location point coordinates, and calculates the sum of each face key point coordinate and the target location point coordinates to obtain multiple target key point coordinates.

[0071] For example: the facial key point coordinates include 3 coordinate points, namely (1,1), (2,2), and (3,3), and the target location point coordinates are (0,1). Then the coordinates of the multiple target key points are (1,2), (2,3), and (3,4).

[0072] By determining the coordinates of the point with the smallest horizontal and vertical coordinate values ​​as the target location point coordinates, the coordinate relationship between the face image and the input image can be established. Then, by combining the target location point coordinates, multiple target key point coordinates can be reasonably generated.

[0073] 105. Based on the coordinates of multiple target key points and preset standard face coordinates, the input image is aligned to obtain an aligned image.

[0074] In at least one embodiment of this application, the preset standard face coordinates refer to the coordinates of key points corresponding to the standard face image. The aligned image refers to the image obtained after facial correction of the face in the input image based on the standard face image. The size of the aligned image can be 40mm*40mm, and in other embodiments, the size of the aligned image can also be 60mm*60mm, and is not limited to this in practical applications.

[0075] In at least one embodiment of this application, the electronic device performs alignment processing on the input image based on the coordinates of multiple target key points and the preset standard face coordinates to obtain an aligned image. This includes: the electronic device calculates a similarity transformation matrix between the input image and the standard face image corresponding to the preset standard face coordinates based on the coordinates of multiple target key points and the preset standard face coordinates; transforms the initial coordinates of each pixel in the input image based on the similarity transformation matrix to obtain the transformed coordinates corresponding to each pixel in the input image; and generates an aligned image based on the pixel points corresponding to the transformed coordinates obtained from the input image.

[0076] The formula for calculating the similarity transformation matrix is: A = M -1 BM, where M represents the similarity transformation matrix, A represents the coordinate matrix corresponding to the coordinates of multiple target key points, and B represents the coordinate matrix corresponding to the preset standard face coordinates. The transformed coordinates can be generated by multiplying the similarity transformation matrix and the coordinate matrix corresponding to the initial coordinates.

[0077] By calculating the similarity transformation matrix between the input image and the standard face image, the face in the input image can be corrected based on the similarity transformation matrix, thereby improving the accuracy of the generated aligned image.

[0078] 106. Based on the face discrimination results of the aligned images, establish a target image library.

[0079] In at least one embodiment of this application, the face discrimination result is used to characterize the degree to which the aligned face meets the preset image requirements. These preset image requirements are set according to the image requirements of the face processing task. The target image library stores multiple images that meet the preset image requirements. The target image library also stores a large number of face background images to support data augmentation for face-related tasks, making the augmented images more natural.

[0080] In at least one embodiment of this application, the electronic device establishes a target image library based on the face discrimination result of the aligned image, including: the electronic device constructs a face discrimination model based on preset image requirements, and discriminates the aligned image based on the face discrimination model to obtain a face discrimination result; if the face discrimination result is greater than a preset confidence level, the mapping information between the aligned image and the coordinates of multiple target key points is stored in the target image library.

[0081] The face recognition model is used to determine how well an image meets preset image requirements. The preset reliability can be set according to actual needs; for example, the preset reliability can be 0.9.

[0082] By using a face discrimination model to discriminate aligned images, the matching degree between the aligned images and the preset image requirements can be obtained. This allows for the filtering of aligned images based on a pre-set reliability, and the construction of a target image library. This improves the rationality of the generation of images stored in the target image library, thereby avoiding the need for preprocessing of aligned images when processing face tasks in the future and improving the effectiveness of the target image library.

[0083] Specifically, the electronic device constructs a face discrimination model based on preset image requirements by: acquiring training images based on preset image requirements, acquiring the matching degree between the training images and the preset image requirements, and adjusting the acquired preset learner based on the training images and the matching degree until the discrimination accuracy of the adjusted learner is greater than or equal to a preset accuracy threshold, thereby obtaining a face discrimination model.

[0084] The preset learner can be any neural network. This application does not impose specific restrictions on the network structure of the preset learner. The preset accuracy threshold can be set according to actual needs.

[0085] Specifically, the electronic device adjusts the acquired preset learner based on the training images and the demand matching degree until the discrimination accuracy of the adjusted learner is greater than or equal to the preset accuracy threshold, thereby obtaining a face discrimination model. This includes: the electronic device discriminates the training images based on the preset learner to obtain a predicted discrimination result, and calculates the ratio of the difference between the predicted discrimination result and the demand matching degree to the number of training images to obtain the discrimination accuracy. If the discrimination accuracy is less than the preset accuracy threshold, the learning parameters in the preset learner are adjusted until the discrimination accuracy is greater than or equal to the preset accuracy threshold, thereby obtaining a face discrimination model.

[0086] By combining the prediction results, the degree of matching requirements, and the number of images, a reasonable discrimination accuracy can be generated, thereby improving the discrimination accuracy of the face discrimination model.

[0087] In other implementations, if the face recognition result is less than or equal to a preset confidence level, it indicates that the preset image requirements are not met, and the electronic device may choose to discard the input image.

[0088] As can be seen from the above technical solutions, this application extracts face images from input images by detecting positions. This not only preserves face information completely but also reduces the space occupied by the face image, while improving the efficiency of key point detection in the face image, thereby improving image preprocessing efficiency. Furthermore, by combining the coordinates of multiple target key points and preset standard face coordinates, the input image is aligned, making the aligned image closer to the standard face image. This avoids the loss of image information caused by different shooting lighting methods and camera angles, improving the robustness of the aligned image. Moreover, by judging faces in the aligned image, the input image can be filtered to remove images that do not meet the face requirements, thereby improving the accuracy of the target image library. In addition, by establishing a target image library, since it is not necessary to repeatedly preprocess the acquired input images each time face-related tasks are processed, the task execution efficiency is improved.

[0089] like Figure 11 The diagram shown is a functional block diagram of a preferred embodiment of the image library construction apparatus in this application. The image library construction apparatus 11 includes a detection unit 110, a cropping unit 111, a generation unit 112, an alignment unit 113, and a building unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0090] The detection unit 110 is used to perform face detection on the acquired input image to obtain the detection position of the face detection box in the input image; the cropping unit 111 is used to crop the face image from the input image based on the detection position; the detection unit 110 is also used to perform face key point detection on the face image to obtain the coordinates of multiple face key points; the generation unit 112 is used to generate multiple target key point coordinates corresponding to the multiple face key point coordinates according to the target position of the face image and the coordinates of multiple face key points; the alignment unit 113 is used to perform alignment processing on the input image according to the coordinates of multiple target key points and preset standard face coordinates to obtain an aligned image; the establishment unit 114 is used to establish a target image library according to the face discrimination result of the aligned image.

[0091] For specific implementation details regarding the functions of each unit in this embodiment, please refer to the following: Figure 2 The relevant descriptions will not be repeated here.

[0092] As can be seen from the above technical solutions, this application extracts face images from input images by detecting positions. This not only preserves face information completely but also reduces the space occupied by the face image, while improving the efficiency of key point detection in the face image, thereby improving image preprocessing efficiency. Furthermore, by combining the coordinates of multiple target key points and preset standard face coordinates, the input image is aligned, making the aligned image closer to the standard face image. This avoids the loss of image information caused by different shooting lighting methods and camera angles, improving the robustness of the aligned image. Moreover, by judging faces in the aligned image, the input image can be filtered to remove images that do not meet the face requirements, thereby improving the accuracy of the target image library. In addition, by establishing a target image library, since it is not necessary to repeatedly preprocess the acquired input images each time face-related tasks are processed, the task execution efficiency is improved.

[0093] like Figure 12 The diagram shown is a schematic representation of the structure of an electronic device that implements the image library construction method in a preferred embodiment of this application.

[0094] In one embodiment of this application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as an image library building program.

[0095] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0096] Processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. Processor 13 is the computational core and control center of electronic device 1, connecting various parts of electronic device 1 through various interfaces and lines, and executing the operating system of electronic device 1, as well as various installed application programs and program code.

[0097] For example, computer-readable instructions can be divided into one or more modules / units, one or more of which are stored in memory 12 and executed by processor 13 to complete this application. One or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer-readable instructions in electronic device 1. For example, the computer-readable instructions can be divided into a detection unit 110, an interception unit 111, a generation unit 112, an alignment unit 113, and an establishment unit 114.

[0098] The memory 12 can be used to store computer-readable instructions and / or modules. The processor 13 implements various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12 and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. The memory 12 may include non-volatile and volatile memory, such as: hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other storage devices.

[0099] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory stick, a TF card (Trans-flash Card), etc.

[0100] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when executed by a processor, the computer-readable instructions can implement the steps of the various method embodiments described above.

[0101] Computer-readable instructions include computer-readable instruction code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer-readable instruction code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), and random access memory (RAM).

[0102] The blockchain referred to in this application is a novel application model of computer technologies such as distributed image library construction, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0103] Combination Figure 2 The memory 12 in the electronic device 1 stores computer-readable instructions, and the processor 13 can execute the computer-readable instructions stored in the memory 12 to achieve, for example... Figure 2 The image library construction method shown.

[0104] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer-readable instructions can be found in [reference]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0106] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0108] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0109] Furthermore, it is clear that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for constructing an image library, characterized in that, The image library construction method includes: The method involves performing face detection on the acquired input image and determining the detection location of face detection boxes in the input image. This includes: acquiring a pre-trained face detection model, which includes a feature extraction network and a detection box generation network. The feature extraction network includes a fine feature extraction layer and a coarse feature extraction layer. The detection box generation network includes a face recognition layer, a target convolutional layer, a target fusion layer, the fine feature extraction layer, and a detection box localization layer. The face recognition layer includes two hierarchical recognition layers. Each hierarchical recognition layer includes three standard convolutional layers, two depthwise separable convolutional layers, two activation layers, one pooling layer, one normalization layer, one fusion layer, and one stacking layer. The first standard convolutional layer in the three standard convolutional layers is connected in series with the first depthwise separable convolutional layer and the pooling layer. The first depthwise separable convolutional layer is connected in series with the first activation layer in the two activation layers. The first activation layer is connected in series with the second standard convolutional layer in the three standard convolutional layers. The pooling layer is connected in series with the two depthwise separable convolutional layers. The system comprises: a second depthwise separable convolution in a depthwise separable convolution; a second depthwise separable convolution connected in series with the second activation layer of the two activation layers; the second activation layer connected in series with the third standard convolution layer of the three standard convolution layers; the third standard convolution layer connected in series with the normalization layer; both the second standard convolution layer and the normalization layer connected in series with the fusion layer; and the fusion layer connected in series with the overlay layer. The system extracts target coarse features from the input image based on the feature extraction network. Based on the face recognition layer, it identifies primary face features from the target coarse features. Using the target convolution layer and the target fusion layer, it generates aggregate features of the primary face features. Based on the fine feature extraction layer, it extracts target precision features from the aggregate features. Based on the face recognition layer, it identifies target face features from the target precision features. Based on the detection box localization layer, it localizes the target face features to obtain the face detection box. The position of the face detection box in the input image is used as the detection position. Based on the detection location, a face image is extracted from the input image; Facial landmark detection is performed on the face image to obtain the coordinates of multiple facial landmarks; Based on the target location of the face image and the coordinates of the multiple facial key points, generate multiple target key point coordinates corresponding to the coordinates of the multiple facial key points; Based on the coordinates of the multiple target key points and the preset standard face coordinates, the input image is aligned to obtain an aligned image; A target image library is established based on the face discrimination results of the aligned images.

2. The image library construction method as described in claim 1, characterized in that, Extracting the face image from the input image based on the detected location includes: Based on the detection location, the initial height and initial width of the face detection box are identified; The initial height is processed based on a preset ratio to obtain the target height, and the initial width is processed based on the preset ratio to obtain the target width; Calculate the height difference between the target height and the initial height, and calculate the width difference between the target width and the target width; The target location is identified based on the height difference, the width difference, and the detection position. The region corresponding to the target location is extracted from the input image to form the face image.

3. The image library construction method as described in claim 2, characterized in that, The detection location includes multiple initial coordinate points, the target location includes multiple target points, and identifying the target location based on the height difference, the width difference, and the detection location includes: The initial coordinate point with the minimum x-coordinate value and the minimum y-coordinate value is determined as the first coordinate point, and the initial coordinate point with the maximum x-coordinate value and the maximum y-coordinate value is determined as the second coordinate point. The first coordinate point includes the first x-coordinate value and the first y-coordinate value, and the second coordinate point includes the second x-coordinate value and the second y-coordinate value. Calculate the width ratio of the width difference to the preset value, and calculate the height ratio of the height difference to the preset value; A target point corresponding to the first coordinate point is generated based on the difference between the first horizontal coordinate value and the width ratio, and the difference between the first vertical coordinate value and the height ratio. A target point corresponding to the second coordinate point is generated based on the sum of the second horizontal coordinate value and the width ratio, and the sum of the second vertical coordinate value and the height ratio.

4. The image library construction method as described in claim 1, characterized in that, The step of detecting facial landmarks in the face image to obtain the coordinates of multiple facial landmarks includes: Obtain a pre-trained keypoint detection model, which includes a feature detection network, a keypoint detection network, a keypoint adjustment network, and a keypoint transformation network. Based on the feature detection network, key point region features are extracted from the face image; Based on the key point detection network, initial key point features are detected from the key point region features; Based on the key point adjustment network, the initial key point features are adjusted to obtain the target key point features; Based on the key point transformation network, the target key point features are transformed to obtain the coordinates of the multiple facial key points.

5. The image library construction method as described in claim 1, characterized in that, The target location includes multiple location point coordinates. Generating multiple target key point coordinates corresponding to the multiple facial key point coordinates based on the target location of the face image and the multiple facial key point coordinates includes: The coordinates of the point with the minimum x-coordinate and minimum y-coordinate values ​​are determined as the coordinates of the target point. The sum of the coordinates of each facial key point and the coordinates of the target location point is calculated to obtain the coordinates of the multiple target key points.

6. The image library construction method as described in claim 1, characterized in that, The step of aligning the input image based on the coordinates of the multiple target key points and the preset standard face coordinates to obtain an aligned image includes: Based on the coordinates of the multiple target key points and the preset standard face coordinates, calculate the similarity transformation matrix between the input image and the standard face image corresponding to the preset standard face coordinates; Based on the similarity transformation matrix, the initial coordinates of each pixel in the input image are transformed to obtain the transformed coordinates corresponding to each pixel in the input image; The aligned image is generated based on the pixels in the input image that correspond to the transformed coordinates.

7. The image library construction method as described in claim 1, characterized in that, The step of establishing the target image library based on the face discrimination results of the aligned images includes: A face recognition model is constructed based on preset image requirements; The aligned image is then analyzed based on the face discrimination model to obtain the face discrimination result. If the face recognition result is greater than the preset confidence level, the mapping information between the aligned image and the coordinates of the multiple target key points is stored in the target image library.

8. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the image library construction method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions that are executed by a processor in an electronic device to implement the image library construction method as described in any one of claims 1 to 7.

Images