Target face image matching method and its device, equipment, medium and product
Through the combination of image segmentation and face detection models, the face area is extracted and aligned, and feature matching technology, the efficiency and accuracy problems when identifying sensitive characters in the existing technology are solved, and efficient and accurate target face recognition is achieved .
Patent Information
- Application Number
- CN202111205309.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-15
AI Technical Summary
When existing neural network models identify sensitive characters including posture, occlusion, and age changes, it is difficult to accurately identify target faces from advertising pictures, resulting in e-commerce platforms facing recognition efficiency and accuracy problems.
The image segmentation model is used to extract human images, and the face detection model is used for face detection. The candidate boxes for segmented pictures are retained for the same face area, the candidate boxes for the image to be detected are deleted, the face alignment and cropping is performed, and the model is extracted through the image feature. Match with the preset face feature library and output similarity results.
It improves the work efficiency and recognition accuracy of the face matching system, reduces background information interference, enhances the success rate of target face image matching, and provides higher confidence detection results.
Smart Images

Figure CN113947792B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a target face image matching method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] Using artificial neural network models for face detection has become a mainstream technology. For application scenarios such as e-commerce platforms, a large number of advertising pictures or other pictures are generated every day. Due to the restrictions of e-commerce platforms on advertising pictures of sensitive people, it is generally necessary to detect and identify the target people in the advertising pictures uploaded by users, and then further process them according to different target people and needs.
[0003] For application scenarios such as e-commerce platforms, if the advertisement images uploaded by merchants to display advertisement information contain target persons, such as persons with political or other sensitive information, the existing neural network models sometimes find it difficult to accurately identify them from the advertisement images due to uncertain factors such as human posture, occlusion, and age changes. This will cause great trouble for e-commerce platforms. It is even more unrealistic to rely entirely on manual investigation.
[0004] Therefore, how to accurately and efficiently identify the target face, that is, the target image, from various types of images to be detected that contain people, so as to make the recognition result more accurate, has become a technical problem that needs to be solved in this field. Summary of the invention
[0005] The primary purpose of the present application is to solve at least one of the above problems and to provide a target face image matching method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.
[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:
[0007] A target face image matching method provided for one of the purposes of this application comprises the following steps:
[0008] Perform face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtain candidate frames and facial key point information corresponding to the face image part of the human image;
[0009] For the candidate frames corresponding to the same face image part, the candidate frames belonging to the segmented image are retained, and the candidate frames belonging to the image to be detected are deleted;
[0010] According to the remaining candidate frames and their associated facial key point information, the facial image parts pointed to by each candidate frame are aligned and cropped to obtain the facial images corresponding to each candidate frame;
[0011] According to the facial feature vector of the facial image, a preset facial feature library is searched to determine a preset facial feature vector similar to the facial feature vector, and the search result is output.
[0012] In a further embodiment, face detection is performed on the image to be detected and the segmented image corresponding to the human image therein, and candidate frames and facial key point information corresponding to the face image portion of the human image are obtained, including the following steps:
[0013] Get the image to be detected;
[0014] Calling a pre-trained image segmentation model to perform image segmentation on the image to be detected, and obtaining a segmented image of the human body image therein;
[0015] The pre-trained face detection model is called to perform face detection on the image to be detected and the segmented image respectively, and the candidate frame and its face key point information corresponding to the face image part in the human body image in the image to be detected and the candidate frame and its face key point information corresponding to the face image part in the human body image in the segmented image are obtained.
[0016] In a specific embodiment, obtaining a picture to be detected includes the following steps:
[0017] Responding to an advertisement publishing request triggered by a user, obtaining the corresponding advertisement publishing information submitted by the user, wherein the advertisement publishing information includes an advertisement picture;
[0018] The advertisement picture is obtained from the advertisement publishing information.
[0019] In a further embodiment, for candidate frames corresponding to the same face image part, the candidate frames belonging to the segmented image are retained, and the candidate frames belonging to the image to be detected are deleted, including the following steps:
[0020] Calculate the intersection-over-union ratio between the candidate frames of the image to be detected and the segmented image based on pixels;
[0021] Comparing the intersection-and-combination score with a preset threshold;
[0022] When the intersection-over-union score between the candidate frame of the image to be detected and the candidate frame in the segmented image is greater than a preset threshold, the candidate frame in the image to be detected is deleted, and the candidate frame in the segmented image is retained.
[0023] In a further embodiment, the facial image portion pointed to by each candidate frame is aligned and cropped according to the remaining candidate frames and the associated facial key point information to obtain the facial image corresponding to each candidate frame, including the following steps:
[0024] Determine whether the remaining candidate frames have corresponding facial key point information;
[0025] When the candidate frame has corresponding facial key point information, determine the transformation matrix corresponding to the facial image from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the facial key point information;
[0026] Align and crop the face image corresponding to the candidate frame from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the transformation matrix;
[0027] When there is no corresponding facial key point information in the candidate frame, the size of the candidate frame is enlarged according to a preset amplitude, and a corresponding image is cropped from the to-be-detected picture or the segmented picture corresponding to the candidate frame as the facial image corresponding to the candidate frame.
[0028] In a further embodiment, searching a preset facial feature library according to the facial feature vector of the facial image to determine a preset facial feature vector similar to the facial feature vector and outputting the search result includes the following operations:
[0029] Calling a pre-trained image feature extraction model to extract a facial feature vector from the facial image;
[0030] Performing similarity calculation on the facial feature vector and a plurality of preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result;
[0031] Arrange the similarities in the similarity calculation results in reverse order, and select some preset face feature vectors with the highest order as similar face feature vectors;
[0032] Output the target face image matching result, which includes the target face image determined according to the uniqueness of the similar face feature vector.
[0033] A target face image matching device provided to meet one of the purposes of the present application includes: a face detection module, a candidate optimization module, a face alignment module and a similarity matching module, wherein the face detection module is used to perform face detection on a to-be-detected image and a segmented image corresponding to a human body image therein, and obtain candidate frames and face key point information corresponding to the face image portion of the human body image; the candidate optimization module is used to retain the candidate frames belonging to the segmented image and delete the candidate frames belonging to the to-be-detected image for the candidate frames corresponding to the same face image portion; the face alignment module is used to align and crop the face image portion pointed to by each candidate frame according to the remaining candidate frames and their associated face key point information, and obtain the face image corresponding to each candidate frame; the similarity matching module is used to search a preset face feature library according to the face feature vector of the face image to determine a preset face feature vector similar to the face feature vector, and output the search result.
[0034] In a further embodiment, the face detection module includes: an image acquisition submodule, used to acquire the image to be detected to be identified as to whether it contains a target person; a human body segmentation submodule, used to call a pre-trained image segmentation model to perform image segmentation on the image to be detected of the image acquisition module, and obtain a segmented image of the human body image; a face detection submodule, used to call a pre-trained face detection model to perform face detection on the image to be detected and the segmented image respectively, and obtain the face candidate frame and key point information of the image to be detected, and the face candidate frame and key point information of the segmented image.
[0035] In a specific example, the image acquisition submodule includes: a request response unit, used to respond to an advertising publishing request triggered by a user and obtain the corresponding advertising publishing information submitted, wherein the advertising publishing information includes advertising images; and an image extraction unit, used to obtain advertising images from the advertising publishing information.
[0036] In a further example, the candidate optimization module includes: a score calculation unit, which is used to calculate the intersection-and-union score between the candidate frames of the image to be detected and the segmented image based on pixels; a comparison unit, which is used to compare the intersection-and-union score with a preset threshold; and a result processing unit, which is used to delete the corresponding candidate frame in the image to be detected and retain the corresponding candidate frame in the segmented image when the intersection-and-union score is greater than the preset threshold.
[0037] In a further example, the face alignment module includes: a judgment unit, used to judge whether the remaining candidate frames have associated facial key point information; a first decision unit, used to determine the transformation matrix corresponding to the facial image from the picture to be detected or the segmented picture corresponding to the candidate frame according to the facial key point information when the candidate frame has corresponding facial key point information, and align and crop the facial image corresponding to the candidate frame from the picture to be detected or the segmented picture corresponding to the candidate frame according to the transformation matrix; a second decision unit, used to expand the size of the candidate frame according to a preset amplitude and crop the corresponding image from the picture to be detected or the segmented picture corresponding to the candidate frame as the facial image corresponding to the candidate frame when the candidate frame does not have corresponding facial key point information.
[0038] In a further example, the similarity matching module includes: a facial feature extraction unit, which is used to call a pre-trained image feature extraction model to extract a facial feature vector from the facial image; a similarity calculation unit, which is used to perform similarity calculation on the facial feature vector and multiple preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result; a similarity optimization unit, which is used to sort the similarities in the similarity calculation result in reverse order and select some of the preset facial feature vectors with the highest ranking as similar facial feature vectors; and a result output unit, which is used to output the target facial image matching result, which includes a target facial image determined according to the uniqueness of the similar facial feature vector.
[0039] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the target face image matching method described in the present application.
[0040] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the target face image matching method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.
[0041] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.
[0042] Compared with the prior art, the advantages of this application are as follows:
[0043] This application calls the image segmentation model to extract a segmented image containing only human images from the image to be detected; calls the face detection model to perform face detection on the image to be detected and the segmented image respectively, and obtains all face candidate frames and their associated face key point information; for the candidate frames pointing to the same face area, retains the corresponding candidate frames in the segmented image, and deletes the corresponding candidate frames in the image to be detected. On the one hand, it prevents work redundancy and improves the work efficiency of the matching system; on the other hand, it can avoid interference from background information and improve the accuracy of the matching results; for the remaining candidate frames, if there is associated face key point information, the person they refer to is The face image is aligned and cropped. If there is no associated facial key point information, it is directly scaled to a standard size to obtain all candidate face images to avoid interference from facial angles and scales. The image feature extraction model is called to extract the face feature vectors one by one. The vector is the representation of the candidate face image in a high-dimensional feature space. The similarity between the candidate face feature vector and all preset face feature vectors in the preset face feature library is calculated, and the highest similarity is taken as the matching value of the candidate face feature vector. Further judgment is made. If it is greater than the set threshold, it is determined to be a successful match, otherwise it is determined to be a failed match, and the matching result is output.
[0044] The present application performs collaborative face detection on the image to be detected and the segmented image. On the one hand, the face image extracted from the segmented image removes the interference of background information relative to the image to be detected, and the face feature vector extracted by the image feature extraction model can better characterize the representation of the face part in the high-dimensional feature space, thereby improving the success rate of the target face image matching. On the other hand, since the segmented image is the result of the image to be detected segmenting the human body contour and "cutting out" the image by the image segmentation model, the face area image in the segmented image is also included in the image to be detected; that is, after the image to be detected and the segmented image are respectively subjected to face detection by the face detection model, there will be a part of the candidate frame and face key point information in their respective detection result information, which will refer to the same face area; the present application is for the candidate frame pointing to the same face area, retains the corresponding candidate frame in the segmented image, deletes the corresponding candidate frame in the image to be detected, avoids extracting the same face image, performing repeated face feature vector extraction and subsequent operations, causing work redundancy, time consumption and other problems, thereby improving the work efficiency of the entire face matching system. In summary, the present application can improve the effect of efficient and accurate recognition of the face matching system.
[0045] The judgment result made by this application on whether the image to be detected contains the image of the target person has a higher confidence level and can be highly trusted. It is suitable for application scenarios such as e-commerce platforms to detect whether there are target images corresponding to the target person in advertising pictures, product pictures, comment pictures, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0047] Figure 1 A flowchart of a typical embodiment of the target face image matching method of the present application;
[0048] Figure 2 The improved U based 2 The structural principle diagram of the image segmentation model of net;
[0049] Figure 3 The figure is a schematic diagram of a logical framework for implementing face alignment of the present application.
[0050] Figure 4 Schematic diagram of the process of face detection in the embodiment of the present application;
[0051] Figure 5 A schematic diagram of the process of fusion optimization of face detection results in an embodiment of the present application;
[0052] Figure 6 A schematic diagram of a process for cropping a face image according to a candidate frame in an embodiment of the present application;
[0053] Figure 7 A schematic diagram of the process of matching facial feature vectors in an embodiment of the present application;
[0054] Figure 8 This is a principle block diagram of the target face image matching device of this application;
[0055] Fig. 9 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION
[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0057] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0059] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.
[0060] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0061] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.
[0062] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.
[0063] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.
[0064] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.
[0065] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.
[0066] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.
[0067] A target face image matching method of the present application can be programmed as a computer program product and deployed in a client or server for execution. The method can be executed by accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface.
[0068] See also Figure 1 In a typical embodiment, the target face image matching method of the present application includes the following steps:
[0069] Step S1100, performing face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtaining candidate frames and facial key point information corresponding to the facial image portion of the human image;
[0070] The picture to be detected generally refers to a picture containing a person's image. In an exemplary application scenario used by this application for auxiliary explanation, the picture to be detected can be various pictures containing person's portraits on an e-commerce platform, such as product pictures, advertising pictures, pictures in product reviews, etc. According to the needs of the e-commerce platform, the person images displayed in the pictures are generally non-sensitive or non-controversial people, and the implementation of this application is to identify the corresponding target person image from the advertising picture, that is, to identify a pre-set person with sensitive or controversial information, and therefore it is necessary to identify the corresponding target person image in the relevant pictures.
[0071] In the application scenario of the e-commerce platform, if it is necessary to obtain the picture to be detected, one implementation method is to receive the input of the e-commerce platform user, especially the input when the merchant instance user configures the advertising information, and use the advertising picture in the advertising information as the picture to be detected. Of course, the picture posted by the consumer in the comment area can also be identified and used as the picture to be detected; in another way, the server of the e-commerce platform can batch process the advertising pictures in the e-commerce platform database in the background, and use these advertising pictures as the pictures to be detected to detect the target face image.
[0072] The segmented picture corresponding to the human body image can be obtained by calling a pre-trained image segmentation model to perform image segmentation on the image to be detected;
[0073] See also Figure 2 This application adopts the U 2 The image segmentation model improved by the net basic network architecture is trained to convergence and used in this application as a pre-processing of the face detection model to be detected.
[0074] U 2 The net model is based on the residual convolution principle and has an encoding path and a decoding path. A multi-level decoder is set on the encoding path, and a multi-level decoder is set on the decoding path accordingly. As for the number of levels of encoders and decoders, it can be flexibly determined by those skilled in the art based on prior knowledge such as experiments or experience. Figure 2The example gives six levels of encoders and five levels of decoders. The main point is that it directly transforms the intermediate feature information obtained by each level of decoder and the last level of encoder into image feature information consistent with the original image specifications after 1*1 convolution kernel transformation, Sigmoid activation function and upsampling. Finally, all image feature information is fused in a cascade fusion manner, and the final mask map is generated after 1*1 convolution kernel transformation and Sigmoid activation function.
[0075] Accordingly, using the U 2 The net model obtains the human mask image data (Mask7) of the image to be detected, and according to the human contour information represented by the mask image data, segments the image defined by the corresponding contour from the original image of the specification to form a segmented image. Compared with the original image to be detected, the segmented image deletes the background image information and only leaves the human image information, which plays a role commonly known as "cutting out".
[0076] It is not difficult to understand that this segmented image can be provided to the face detection model for further detection in this application to reflect the role of the image segmentation model in preprocessing the image to be detected.
[0077] After “cutting out”, the segmented image can be obtained, and the segmented image and the image to be detected can be provided to the face detection model for further face detection.
[0078] Those skilled in the art can understand that by pre-training the face detection model, it can learn the ability to detect the target face area information from the image to be detected. Therefore, a large number of images containing various forms of the target face can be used to train the face detection model, so that such a face detection model can learn the ability to recognize the face area information from the advertising image. In other application scenarios, a face detection model for recognizing the target face area information from the image to be detected can also be prepared based on this principle.
[0079] Since the face detection model is a relatively mature technology, there are various face detection models in the prior art that can be used for reference. Generally, they are implemented based on CNN convolutional neural networks, including but not limited to MTCNN, FaceNet, DeepFace, DeepID, etc. Different such neural network-based face detection models may have different convolution structures, resulting in different recognition methods. However, in terms of function and purpose, as long as the face detection model has been pre-trained to a convergence state for the target person and put into the production stage, it can be used to recognize the target person image from the image to be detected. As long as it is trained to convergence with a sufficient amount of corresponding training samples, it can be used as the face detection model of this application. Of course, the recognition capabilities of various models present different recognition accuracy rates depending on the different model structures and training samples, which is reflected in the different confidence levels of the recognition results of each recognition model. In this application, MTCNN is recommended because its actual measured performance is relatively excellent.
[0080] When the face detection model is trained to convergence, face detection can be performed based on the image to be detected and the segmented image respectively to obtain the face candidate frame information in the image to be detected and the face key point information corresponding to each face candidate frame, and the face candidate frame information in the segmented image and the face key point information corresponding to each face candidate frame.
[0081] The facial key points in the detection results of the exemplary MTCNN model include five key position information of the left and right eyes, nose, and two corners of the mouth in the face area, which can be used for subsequent face alignment operations to calculate the similarity transformation matrix. The number of facial key points can be manually set according to actual needs and the model used. In the subsequent description of this application, the five facial key point information extracted by the MTCNN model is used by default.
[0082] Step S1200: for the candidate frames corresponding to the same face image part, retain the candidate frames belonging to the segmented image, and delete the candidate frames belonging to the image to be detected:
[0083] As described above, after step S1100, the face candidate frame information of the image to be detected and the segmented image and the face key point information corresponding to each candidate frame are obtained at the same time.
[0084] Since the segmented image is the result of segmenting the human body contour from the image to be detected by the image segmentation model and performing "cutting", the human body area image in the segmented image is also included in the image to be detected. On this basis, the results of face detection by the face detection model for the segmented image and the image to be detected are very likely to contain the detection result information representing the same face area. In order to avoid the problems of redundant work and large time consumption caused by extracting face feature vectors and subsequent operations for the same face image, it is necessary to perform "de-duplication" operation on the detection result information that can refer to the same face area in the detection results of the segmented image and the image to be detected, so as to improve the working efficiency of the entire face matching system. In this application, when the detection result information referring to the same face area appears, the corresponding detection result information in the image to be detected is selected to be removed, and the corresponding detection result information in the segmented image is retained; the reason is that the segmented image is the result after "cutting", and the interference of background information is removed relative to the image to be detected. In the subsequent extraction of face feature vectors, it can better represent the representation of the face image in the high-dimensional feature space, and improve the success rate of matching the target face image.
[0085] Step S1300: Align and crop the facial image portion pointed to by each candidate frame according to the remaining candidate frames and their associated facial key point information to obtain the facial image corresponding to each candidate frame:
[0086] After step S1200 "deduplication", the remaining detection result information includes candidate frame information and facial key point information associated with the candidate frame, which refers to different facial regions; the remaining detection result information includes both the image to be detected and the segmented image. Figure 3 As shown, the remaining candidate frame information is traversed one by one, the face area referred to by it is framed, and the face alignment is performed according to the associated face key point information to obtain a face image in a standard posture, and then the aligned face image is cropped to a fixed size, and finally multiple candidate standard face images are obtained for subsequent extraction of face feature vectors. The standard face image avoids the interference of face angle and face image scale, and as the standard input of the image feature extraction model, it can extract a face feature vector that can better characterize its face image.
[0087] Step S1400: Search a preset facial feature library according to the facial feature vector of the facial image to determine a preset facial feature vector similar to the facial feature vector, and output the search result:
[0088] After the above steps, all candidate face images are obtained. After image preprocessing, the pre-trained image feature extraction model is called to extract the face feature vector from the face image.
[0089] The image feature extraction model can be selected from a variety of relatively excellent image feature extraction models in the prior art, including but not limited to: VGG series models, Inception series models, ResNet series models, EfficientNet series models, etc., which are all mature image feature extraction models. As long as they are trained to convergence with sufficient corresponding training samples, they can be used as the image feature extraction model of this application. In this application, it is recommended to use the ResNet101 model because its actual measured performance is relatively excellent.
[0090] When the image feature extraction model is trained to convergence, the candidate face image can be input to extract its face feature vector. The face feature vector is the representation of the face image in the high-dimensional feature space. The exact dimension of the high-dimensional feature space can be set as needed. In this application, 512 dimensions are used as an example for illustration.
[0091] The facial feature vector is similarly calculated with multiple preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result between the facial feature vector and each preset facial feature vector; the multiple preset facial feature vectors in the preset facial feature library refer to facial feature vectors of all preset target persons extracted by a pre-trained image feature extraction model, and the pre-trained image feature extraction model is consistent with the image feature extraction model for extracting the candidate facial feature vector, and the consistency includes pre-trained weights and other settings, otherwise the representations of the facial feature vectors extracted by the two in the high-dimensional feature space will be inconsistent, which may affect subsequent matching work.
[0092] After the above processing, a search algorithm is used to search for some preset facial feature vectors with high similarity to the facial feature vectors of the facial image in the image to be detected or the segmented image as similar facial feature vectors, and the similarity value with the highest similarity is sorted to find out as the matching value of the facial image. The matching value represents the degree of matching between the person referred to in the facial image and the target person library referred to in the preset facial feature vector library. A high degree of matching indicates that the person referred to in the facial image is more likely to belong to the target person library, and a low degree of matching indicates that the person referred to in the facial image is more likely to not belong to the target person library. Therefore, the final decision of facial matching is made based on the matching value.
[0093] If the matching value is greater than the preset threshold, that is, the match is successful, then it is determined that the image to be detected contains the target person image; if the calculated matching value is not greater than the preset threshold, that is, the match fails, then it is determined that the image to be detected does not contain the target person image. In the example application scenario, further business processing can be performed on the image to be detected according to the matching situation. The preset threshold is the dividing value for determining whether the matching value is a successful match or not. The setting of this value directly affects the accuracy of the match. If it is set too high, the candidate face feature vector that does not belong to the preset face feature library is judged to be a successful match. If it is set too low, the candidate face feature vector that belongs to the preset face feature library is judged to be a failed match; therefore, the threshold needs to be set by relevant technical personnel based on the comparison of test results and the use of prior knowledge.
[0094] The search algorithm may be selected from a variety of relatively excellent search algorithms in the prior art, including but not limited to Dijkstra algorithm, Kruskal algorithm, Prim algorithm, etc.: In this application, the nearest neighbor search algorithm NNS is recommended.
[0095] It can be seen from this typical embodiment that the present application calls the image segmentation model to extract a segmented image containing only human images from the image to be detected; calls the face detection model to perform face detection on the image to be detected and the segmented image respectively, and obtains all face candidate frames and their associated face key point information; for the candidate frames pointing to the same face area, retains the corresponding candidate frames in the segmented image, and deletes the corresponding candidate frames in the image to be detected. On the one hand, it prevents work redundancy and improves the work efficiency of the matching system; on the other hand, it can avoid interference from background information and improve the accuracy of the matching results; for the remaining candidate frames, if there is associated face key point information, then The face image it refers to is aligned and cropped. If there is no associated face key point information, it is directly scaled to a standard size to obtain all candidate face images and avoid interference from face angles and scales. The image feature extraction model is called to extract the face feature vectors one by one. The vector is the representation of the candidate face image in a high-dimensional feature space. The similarity between the candidate face feature vector and all preset face feature vectors in the preset face feature library is calculated, and the highest similarity is taken as the matching value of the candidate face feature vector for further judgment. If it is greater than the set threshold, it is determined to be a successful match, otherwise it is determined to be a failed match, and the matching result is output.
[0096] The present application performs collaborative face detection on the image to be detected and the segmented image. On the one hand, the face image extracted from the segmented image removes the interference of background information relative to the image to be detected, and the face feature vector extracted by the image feature extraction model can better characterize the representation of the face part in the high-dimensional feature space, thereby improving the success rate of the target face image matching. On the other hand, since the segmented image is the result of the image to be detected segmenting the human body contour and "cutting out" the image by the image segmentation model, the face image in the segmented image is also included in the image to be detected; that is, after the image to be detected and the segmented image are respectively subjected to face detection by the face detection model, there will be a part of the candidate frame and face key point information in their respective detection result information, which will refer to the same face area; the present application is for the candidate frame pointing to the same face area, retains the corresponding candidate frame in the segmented image, deletes the corresponding candidate frame in the image to be detected, avoids extracting the same face image, performing repeated face feature vector extraction and subsequent operations, causing problems such as work redundancy and time consumption, thereby improving the work efficiency of the entire face matching system. In summary, the present application can improve the effect of efficient and accurate recognition of the face matching system.
[0097] The judgment result made by this application on whether the target person is contained in the detected image has a higher confidence level and can be highly trusted. It is suitable for application scenarios such as e-commerce platforms to detect whether there are target images corresponding to the target person in advertising images, product images, comment images, etc.
[0098] See also Figure 4 In a further embodiment, the step S1100, performing face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtaining the candidate frame and face key point information corresponding to the face image part of the human image, comprises the following steps:
[0099] Step S1110, obtaining a picture to be detected;
[0100] As mentioned above, the acquisition of the image to be detected can be extracted from the product information in the product database, or can be received from the user input. In an embodiment of the latter, the advertisement publishing request triggered by the user can be responded to first, and the corresponding submitted advertisement publishing information can be obtained, and the advertisement publishing information includes the advertisement image; then, the advertisement image is obtained from the advertisement publishing information, thereby realizing the timely detection of the image uploaded by the user.
[0101] Step S1120, calling a pre-trained image segmentation model to perform image segmentation on the image to be detected, and obtaining a segmented image of the human body image therein;
[0102] Please review again Figure 2Combined with the above, the image segmentation model is constructed with an encoding path and a decoding path. A multi-level decoder is set on the encoding path, and a multi-level decoder is set on the decoding path accordingly. As for the number of levels of encoders and decoders, it can be flexibly determined by those skilled in the art based on prior knowledge such as experiments and experience. Figure 2 The example gives six levels of encoders and five levels of decoders. The main point is that it directly transforms the intermediate feature information obtained by each level of decoder and the last level of encoder into image feature information consistent with the original image specifications after 1*1 convolution kernel transformation, Sigmoid activation function and upsampling. Finally, all image feature information is fused in a cascade fusion manner, and the final mask map is generated after 1*1 convolution kernel transformation and Sigmoid activation function.
[0103] The encoding path, in this embodiment, Figure 2 The six encoders (En_1 to En_6) in the side branch path convert the original image of the image to be detected into the original image of the specification corresponding to the image to be detected, that is, the image adapted to U 2 net's need for the specifications of the input image and cuts the image to be detected into the image to be detected of the specified specifications, and encodes the original image of this specification level by level. The first-level encoder at the top level extracts the intermediate feature information corresponding to the first scale from the original image of the specification, and then passes it to the encoder at the next level to extract the intermediate feature information corresponding to the second scale. By analogy, the six-level encoder extracts the corresponding intermediate feature information. It can be seen that after the original image of the specification is encoded level by level through the encoding path, six intermediate feature information of corresponding spatial resolution can be obtained.
[0104] It can be understood that the intermediate feature information of each scale is a representation obtained after deep semantic understanding of the standard image at the corresponding scale, and is the information extracted from the contour features of the human body in the image to be detected. 2 This capability of the net model is well known to those skilled in the art. As long as it is trained to convergence with a sufficient number of training samples, its encoding path can have the deep semantic understanding capability to capture the contour features of the human body in the image to be detected.
[0105] The decoding path, in this embodiment, Figure 2In the five decoders (De_1 to De_5) of the right branch path, starting from the bottom layer, each decoder stage takes the upsampled feature map from the previous stage and the cascade of the feature map from its symmetric encoder stage as input, and outputs feature information after decoding. The output results of the last encoder stage and each level of decoder stage are subjected to 1*1 convolution kernel transformation, Sigmoid activation function and upsampling operation to extract six image feature information of the same specifications as the original image. The image feature information can characterize the human body contour information in the image to be detected, and the contour features are represented in the form of masks, which are essentially mask image data (Mask1 to Mask6).
[0106] So far, according to the disclosure of the structure and principle of the image segmentation model herein, it can be understood that the image to be detected in the present application is encoded and decoded step by step by the image segmentation model to obtain multiple image feature information (mask image data: Mask1 to Mask6). 2 net model principle, and finally fuse the image feature information in a cascade fusion manner, and generate the final mask map (Mask7) after 1*1 convolution kernel transformation and Sigmoid activation function.
[0107] U 2 In the net model, multiple image feature information is fused and processed into a single mask image data (Mask7). This mask image data is essentially a binary grayscale image. Figuratively speaking, when there is no human body in the advertising image, the grayscale image is a pure white image; when there is a human body in the advertising image, the grayscale image is a black area defined by the outline of the human body in the foreground and a pure white background. It is not difficult to understand that after fusion, a mask image commonly known as "cutout" is obtained.
[0108] After obtaining the corresponding mask image data to form the mask required for image segmentation, the mask data can be used to extract the image of the area occupied by the human body from the original image of the image to be detected, that is, the image corresponding to the area covered by the black background of the mask, as the segmented image. To this end, the following formula can be used:
[0109] Output Img =mask*Img Org +(1-mask)*255
[0110] Among them, Output Img is a human body segmentation map, and mask refers to the mask image output after the image segmentation model is fully connected ( Figure 2 mask7), Img Org It is the original image of the image to be detected.
[0111] This embodiment further demonstrates the detailed process of the image segmentation model processing the image to be detected, as well as various alternative embodiments thereof. It can be seen that the image segmentation model of the present application is suitable for using the U-based 2 NET variants, not limited to U 2 net version, and can be extended to be compatible with other models with equivalent graph segmentation capabilities.
[0112] Step S1130, calling the pre-trained face detection model to perform face detection on the image to be detected and the segmented image respectively, and obtaining the candidate frame and its face key point information corresponding to the face image part in the human body image in the image to be detected, and the candidate frame and its face key point information corresponding to the face image part in the human body image in the segmented image:
[0113] As mentioned above, after “cutting out”, the segmented image can be obtained, and the segmented image and the image to be detected can be provided to the face detection model for further target face detection.
[0114] As mentioned above, the face detection model can be selected from a variety of relatively excellent face detection models in the prior art, including but not limited to: MTCNN, FaceNet, DeepFace, DeepID and their upgraded and evolved versions, etc., which are all mature face detection models. As long as they are trained to convergence with sufficient corresponding training samples, they can be used as the face detection model of this application. In this application, MTCNN is recommended because of its excellent performance.
[0115] When the face detection model (MTCNN) is trained to convergence, face detection can be performed based on the image to be detected and the segmented image respectively to obtain the face candidate frame information in the image to be detected and the face key point information corresponding to each face candidate frame, and the face candidate frame information in the segmented image and the face key point information corresponding to each face candidate frame.
[0116] The facial key points in the detection results of the MTCNN model include five key position information of the left and right eyes, nose, and two corners of the mouth in the face area, which can be used for subsequent face alignment operations to calculate the similarity transformation matrix. The number of facial key points can be manually set according to actual needs and the model used. This application uses the five facial key point information extracted by the MTCNN model by default.
[0117] See also Figure 5 In a further embodiment, the step S1200, for the candidate frames corresponding to the same face image part, retains the candidate frames belonging to the segmented image, and deletes the candidate frames belonging to the image to be detected, comprises the following steps:
[0118] Step S1210, calculating the intersection-over-union ratio between the candidate frames of the image to be detected and the segmented image based on pixels;
[0119] Since the segmented picture is the result of segmenting the human body contour from the picture to be detected by the image segmentation model and performing "cutting", the human body area image in the segmented picture is also included in the picture to be detected. Therefore, after the segmented picture and the picture to be detected are respectively subjected to face detection by the face detection model, there will be a part of candidate boxes and face key point information in their respective detection result information, which will most likely refer to the same face area. In order to avoid extracting the same face image, performing repetitive face feature vector extraction and subsequent operations, causing problems such as redundant work and high time consumption, the present application will perform a "de-duplication" operation on the detection result information that refers to the same face area in the detection results of the segmented picture and the picture to be detected, thereby improving the working efficiency of the entire face matching system.
[0120] The intersection-over-union value is commonly known as the IOU value, which is the ratio of the intersection and union of the areas of two candidate boxes, and can characterize the correlation between the areas of the objects referred to by the two candidate boxes. The larger the value, the higher the correlation, and vice versa. The present application calculates the intersection-over-union value of any candidate box in the image to be detected and any candidate box in the segmented image based on pixel calculation, that is, the IOU value. The larger the value, the higher the similarity between the face areas referred to by the candidate box from the image to be detected and the candidate box from the segmented image, and vice versa.
[0121] Step S1220, comparing the intersection-and-combination ratio value with a preset threshold;
[0122] According to the above, the intersection-over-union (IOU) score reflects the similarity of the face regions referred to by the two candidate frames; however, in practical applications, only the binary results of "same" and "different" need to be applied. Therefore, a threshold needs to be set as the similarity dividing value. When the IOU value is greater than this value, the two candidate frames are judged to refer to the same face region, otherwise they are judged to be two different face regions.
[0123] The setting of the threshold plays a key role and can indirectly affect the matching results and the working efficiency of the matching system. If the value is set too high, the originally identical face areas will be judged as "different", causing subsequent redundant work problems; if the value is set too low, the originally different face areas will be judged as "same" and then deleted. If the person referred to by the removed face area is the only target person in the image to be detected, it will lead to the presentation of wrong matching results. Therefore, the setting of the threshold needs to be reasonably set by relevant technical personnel through experimental observation, result analysis, and based on prior knowledge such as experience.
[0124] Step S1230: When the intersection-over-union ratio between the candidate frame of the image to be detected and the candidate frame in the segmented image is greater than a preset threshold, the candidate frame in the image to be detected is deleted, and the candidate frame in the segmented image is retained.
[0125] As described above, when the intersection-and-union ratio value is greater than the preset threshold, it indicates that the two candidate boxes refer to the "same" face area. In order to avoid subsequent duplication of work, it is necessary to delete one of the candidate box information and its associated face key point information; when the value is not greater than the preset threshold, it indicates that the two candidate boxes point to two different face areas, and the corresponding candidate box information and its face key point information in the detection results of the image to be detected and the segmented image are retained.
[0126] In the "same" face area of the image to be detected and the segmented image, the corresponding area in the image to be detected contains background information, while the corresponding area in the segmented image does not contain background information. The background information will "interfere" with the subsequent extraction of the face feature vector of the face image to a certain extent, that is, the extracted face feature vector contains the "interference" of the background information, which affects the subsequent matching results. Therefore, when the intersection-and-union ratio value is greater than the preset threshold, the present application chooses to delete the corresponding candidate frame information and its face key point information in the detection result of the image to be detected, and retain the corresponding candidate frame information and its face key point information in the detection result of the segmented image, so as to avoid the interference of the background on the extraction of the face feature vector, thereby improving the matching accuracy.
[0127] Deleting the candidate frame corresponding to the image to be detected and overlapping with the candidate frame corresponding to the segmented image and its associated facial key point information has two main beneficial effects on the entire matching system. First, it removes repeated facial areas, avoids extracting the same facial image, performing repeated facial feature vector extraction and subsequent operations, and causes problems such as redundant work and time consumption. This application performs a "de-duplication" operation on the detection result information referring to the same facial area in the detection results of the segmented image and the image to be detected, thereby improving the work efficiency of the entire face matching system; second, in the facial image determined to be "the same", the corresponding area in the image to be detected contains background information, and the corresponding area in the segmented image does not contain background information. The facial feature vector extracted by the image feature extraction model avoids the interference of background information, making it more able to characterize the representation of the facial image in the high-dimensional feature space, thereby improving the success rate of matching the target facial image. It can be seen that the "de-duplication" operation can improve the effect of efficient and accurate recognition of the face matching system.
[0128] See also Figure 6In the further example, the step S1300 aligns and crops the facial image portion pointed to by each candidate frame according to the remaining candidate frames and the associated facial key point information to obtain the facial image corresponding to each candidate frame, including the following steps:
[0129] Step S1310, determining whether the remaining candidate frames have corresponding facial key point information;
[0130] According to the remaining candidate frames and their associated facial key point information results after processing in step S1200, since the output results of the face detection model generally include the detection frame information and its associated facial key point information, but there may also be a situation where only the detection frame information is included without its associated facial key point information, and the facial key point information can be applied to the "face alignment" operation to reduce the interference of the face angle, it is necessary to distinguish these two situations and process them separately. Therefore, each candidate frame in the detection result information is traversed to determine whether the candidate frame has corresponding facial key point information for further processing.
[0131] Step S1320: when the candidate frame has corresponding facial key point information, determine the transformation matrix corresponding to the facial image from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the facial key point information;
[0132] Some of the remaining candidate box information belongs to the image to be detected, and some belongs to the segmented image. The candidate box refers to the corresponding face area of the image.
[0133] When the candidate frame has corresponding facial key point information, the face alignment algorithm can be used to solve the transformation matrix based on the facial key point information of the face image referred to by the current candidate frame and the standard facial key point information. The transformation matrix refers to the similarity transformation operation, which is composed of translation transformation, scale transformation and rotation transformation. It can transform the face image in any posture before alignment into a face image in standard posture.
[0134] In this application, a variety of relatively excellent face alignment algorithms in the existing technologies can be selected, such as the active appearance model (AAM), the cascade regression method, the Constrained Local Model, etc. This application uses MTCNN to obtain facial key point information, and then uses the Umeyama alignment algorithm to obtain the similarity transformation matrix corresponding to the face image before alignment and the face image after alignment.
[0135] Step S1330, aligning and cropping the face image corresponding to the candidate frame from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the transformation matrix;
[0136] According to the transformation matrix of step S1320, the original face image in any posture is transformed similarly to obtain the face image in the standard posture. This process is the "face alignment" operation, which can avoid the interference of different face directions on the subsequent extraction of face feature vectors. The face images after "face alignment" have different sizes, and the image feature extraction model with standard input cannot be directly called; therefore, it is necessary to crop the aligned face image according to the standard input size preset in the image feature extraction model. The standard input size can be set according to the actual test results. The cropping refers to adjusting the size along the shortest dimension of the width and height of the image, and then cropping the image with the center as the origin to obtain the corresponding face image.
[0137] Step S1340: When the candidate frame does not have corresponding facial key point information, the size of the candidate frame is enlarged according to a preset amplitude, and a corresponding image is cropped from the to-be-detected picture or the segmented picture corresponding to the candidate frame as the facial image corresponding to the candidate frame:
[0138] The preset amplitude is the standard input size of the image feature extraction model. When the candidate frame does not have associated facial key point information, the original face image does not need to undergo the "face alignment" operation, and the image can be cropped according to the preset amplitude to directly match the standard input size of the image feature extraction model.
[0139] This embodiment aligns and crops the facial image portion pointed to by the candidate frame according to the remaining candidate frames and their associated facial key point information to obtain the facial images corresponding to each candidate frame. For the entire matching system, the interference of the facial angle and size in the facial image on the subsequent facial feature vector extraction is avoided, the facial feature vector is guaranteed to represent the person in the high-dimensional feature space, and the accuracy of the matching results is improved.
[0140] See also Figure 7 In the in-depth example, step S1400, searching a preset face feature library according to the face feature vector of the face image to determine a preset face feature vector similar to the face feature vector, and outputting the search result, comprises the following steps:
[0141] Step S1410, calling a pre-trained image feature extraction model to extract a facial feature vector from the facial image;
[0142] After the above steps, all candidate face images are obtained, and image preprocessing operations are performed on them, such as image normalization processing, etc. The specific processing method can be set according to the needs of relevant technical personnel in the corresponding scenario, and then the pre-trained image feature extraction model is called to extract the candidate face feature vectors from all the candidate face images one by one.
[0143] The image feature extraction model can select a variety of relatively excellent image feature extraction models in the prior art, including but not limited to: VGG series models, Inception series models, ResNet series models, EfficentNet series models, etc., which are all mature image feature extraction models. As long as they are trained to convergence with sufficient corresponding training samples, they can be used as the image feature extraction model of this application.
[0144] When the image feature extraction model is trained to convergence, a facial feature vector representation in a high-dimensional feature space can be extracted based on the facial image, which can characterize the key feature information of the facial image. The exact dimension of the facial feature vector can be set according to the needs of relevant technicians in the corresponding scenario.
[0145] Step S1420, performing similarity calculation on the facial feature vector and a plurality of preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result;
[0146] The similarity calculation is performed on all candidate facial feature vectors one by one with multiple preset facial feature vectors in the preset facial feature library to obtain similarity calculation results between them and each preset facial feature vector. The multiple preset facial feature vectors in the preset facial feature library refer to facial feature vectors extracted from facial images of all preset target persons by a pre-trained image feature extraction model, and the pre-trained image feature extraction model is consistent with the image feature extraction model for extracting candidate facial feature vectors, and the consistency includes pre-trained weights and other settings, otherwise the representation of the facial feature vectors extracted by the two in the high-dimensional feature space will be inconsistent, affecting subsequent matching work.
[0147] In this application, a variety of different "distance" similarity calculation methods can be used, such as Euclidean distance, Manhattan distance, angle cosine, etc. In addition, it is also recommended to use the FAISS framework to implement the construction of the similarity matrix and similarity query matching. The framework provides the relevant interfaces required to implement the entire process of similarity calculation, which can simplify the implementation process of this application.
[0148] Step S1430, sorting the similarities in the similarity calculation results in reverse order, and selecting some preset face feature vectors with the highest order as similar face feature vectors;
[0149] After the above processing, a search algorithm is used to search for some preset face feature vectors with high similarity to the face feature vectors of the face image in the image to be detected or the segmented image as similar face feature vectors. The reverse order arrangement is to arrange the similarity in descending order from large to small, that is, the opposite of the ascending order, so that the large similarity value is arranged in the front, and the small similarity value is arranged in the back, and finally the front part of the preset face feature vectors is directly selected as the similar face feature vector;
[0150] As mentioned above, the search algorithm may be selected from a variety of relatively excellent search algorithms in the prior art, including but not limited to Dijkstra algorithm, Kruskal algorithm, Prim algorithm, etc.: In this application, the nearest neighbor search algorithm NNS is recommended.
[0151] Step S1440: Output the target face image matching result, which includes the target face image determined according to the uniqueness of the similar face feature vector:
[0152] Find the similarity value with the highest similarity in the similar facial feature vector as the matching value of the candidate facial image. This value represents the degree of matching between the person referred to in the facial image and the target person library referred to in the preset facial feature vector library. A high degree of matching indicates that the person referred to in the facial image is more likely to belong to the target person library, and a low degree of matching indicates that the person referred to in the facial image is more likely not to belong to the target person library. Therefore, the final decision of facial matching is made based on the matching value.
[0153] If the match value is greater than the preset threshold, that is, the match is successful, then it is determined that the image to be detected contains the target person; if the calculated match value is not greater than the preset threshold, that is, the match fails, then it is determined that the image to be detected does not contain the target person. In the example application scenario, further business processing can be performed on the image to be detected according to the matching situation. The preset threshold is the dividing value for determining whether the match value is a successful match or not. The setting of this value directly affects the accuracy of the match. If it is set too high, the candidate face feature vector that does not belong to the preset face feature library is judged to be a successful match. If it is set too low, the candidate face feature vector that belongs to the preset face feature library is judged to be a failed match; therefore, the threshold needs to be set by relevant technical personnel based on the comparison of test results and the use of prior knowledge.
[0154] In a preferred embodiment, the image segmentation model is U 2 net model, the face detection model is the MTCNN model, the feature extraction model is the ResNet101 model, and the basic network architecture of the image segmentation model is the U 2net model, the basic network architecture of the face detection model is the MTCNN model, the basic network architecture of the feature extraction model is the ResNet101 model, and the target person is a sensitive or controversial person, such as an unauthorized celebrity, an authoritative person, etc.
[0155] See also Figure 8 , a target face image matching device provided to meet one of the purposes of the present application, comprising: a face detection module 1100, a candidate optimization module 1200, a face alignment module 1300 and a similar matching module 1400, wherein the face detection module 1100 is used to perform face detection on a to-be-detected image and a segmented image corresponding to a human body image therein, and obtain candidate frames and face key point information corresponding to the face image portion of the human body image; the candidate optimization module 1200 is used to retain the candidate frames belonging to the segmented image and delete the candidate frames belonging to the to-be-detected image for the candidate frames corresponding to the same face image portion; the face alignment module 1300 is used to align and crop the face image portion pointed to by each candidate frame according to the remaining candidate frames and their associated face key point information, and obtain the face image corresponding to each candidate frame; the similar matching module 1400 is used to search a preset face feature library according to the face feature vector of the face image to determine a preset face feature vector similar to the face feature vector, and output the search result.
[0156] In a further embodiment, the face detection module 1100 includes: a picture acquisition submodule, used to acquire a picture to be detected to be identified as to whether it contains a target person; a human body segmentation submodule, used to call a pre-trained image segmentation model to perform image segmentation on the picture to be detected of the picture acquisition module, and obtain a segmented picture of the human body image; a face detection submodule, used to call a pre-trained face detection model to perform face detection on the picture to be detected and the segmented picture respectively, and obtain the face candidate frame and key point information of the picture to be detected, and the face candidate frame and key point information of the segmented picture.
[0157] In a specific example, the image acquisition submodule includes: a request response unit, used to respond to an advertising publishing request triggered by a user and obtain the corresponding advertising publishing information submitted, wherein the advertising publishing information includes advertising images; and an image extraction unit, used to obtain advertising images from the advertising publishing information.
[0158] In the further example, the candidate optimization module 1200 includes: a score calculation unit, which is used to calculate the intersection-and-union score between the candidate frames of the image to be detected and the segmented image based on pixels; a comparison unit, which is used to compare the intersection-and-union score with a preset threshold; and a result processing unit, which is used to delete the corresponding candidate frame in the image to be detected and retain the corresponding candidate frame in the segmented image when the intersection-and-union score is greater than the preset threshold.
[0159] In the further example, the face alignment module 1300 includes: a judgment unit, used to judge whether the remaining candidate frames have associated facial key point information; a first decision unit, used to determine the transformation matrix corresponding to the facial image from the picture to be detected or the segmented picture corresponding to the candidate frame according to the facial key point information when the candidate frame has corresponding facial key point information, and align and crop the facial image corresponding to the candidate frame from the picture to be detected or the segmented picture corresponding to the candidate frame according to the transformation matrix; a second decision unit, used to expand the size of the candidate frame according to a preset amplitude and crop the corresponding image from the picture to be detected or the segmented picture corresponding to the candidate frame as the facial image corresponding to the candidate frame when the candidate frame does not have corresponding facial key point information.
[0160] In the further example, the similarity matching module 1400 includes: a facial feature extraction unit, which is used to call a pre-trained image feature extraction model to extract a facial feature vector from the facial image; a similarity calculation unit, which is used to perform similarity calculation on the facial feature vector and multiple preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result; a similarity optimization unit, which is used to sort the similarities in the similarity calculation result in reverse order, and select some of the preset facial feature vectors with the highest ranking as similar facial feature vectors; and a result output unit, which is used to output the target facial image matching result, which includes the target facial image determined according to the uniqueness of the similar facial feature vector.
[0161] In order to solve the above technical problems, the present application also provides a computer device. Fig. 9As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a target face image matching method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the target face image matching method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0162] In this embodiment, the processor is used to execute Figure 8 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the target face image matching device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.
[0163] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the target face image matching method of any embodiment of the present application.
[0164] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.
[0165] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0166] In summary, the present application can more accurately and efficiently identify relevant controversial or sensitive figures from the images to be detected, and is suitable for target face image recognition of product images on e-commerce platforms, and its detection results have a high degree of confidence.
[0167] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0168] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A target face image matching method, characterized in that: The steps include: Perform face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtain candidate frames and facial key point information corresponding to the face image part of the human image; For candidate frames corresponding to the same face image part, retain the candidate frames belonging to the segmented picture, and delete the candidate frames belonging to the picture to be detected, including: calculating the intersection-and-union score between the candidate frames of the picture to be detected and the segmented picture based on pixels; comparing the intersection-and-union score with a preset threshold; when the intersection-and-union score between the candidate frame of the picture to be detected and the candidate frame in the segmented picture is greater than the preset threshold, deleting the candidate frame in the picture to be detected, and retaining the candidate frame in the segmented picture; According to the remaining candidate frames and their associated facial key point information, the facial image parts pointed to by each candidate frame are aligned and cropped to obtain the facial images corresponding to each candidate frame; According to the facial feature vector of the facial image, a preset facial feature library is searched to determine a preset facial feature vector similar to the facial feature vector, and the search result is output.
2. The target face image matching method according to claim 1, characterized in that: Performing face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtaining candidate frames and face key point information corresponding to the face image part of the human image, comprises the following steps: Get the image to be detected; Calling a pre-trained image segmentation model to perform image segmentation on the image to be detected, and obtaining a segmented image of the human body image therein; The pre-trained face detection model is called to perform face detection on the image to be detected and the segmented image respectively, and the candidate frame and its face key point information corresponding to the face image part in the human body image in the image to be detected and the candidate frame and its face key point information corresponding to the face image part in the human body image in the segmented image are obtained.
3. The target face image matching method according to claim 2, characterized in that: Obtaining the image to be detected includes the following steps: Responding to an advertisement publishing request triggered by a user, obtaining the corresponding advertisement publishing information submitted by the user, wherein the advertisement publishing information includes an advertisement picture; The advertisement picture is obtained from the advertisement publishing information.
4. The target face image matching method according to claim 1, characterized in that: For candidate frames corresponding to the same face image part, retain the candidate frames belonging to the segmented image, and delete the candidate frames belonging to the image to be detected, including the following steps: Calculate the intersection-over-union ratio between the candidate frames of the image to be detected and the segmented image based on pixels; Comparing the intersection-and-combination score with a preset threshold; When the intersection-over-union score between the candidate frame of the image to be detected and the candidate frame in the segmented image is greater than a preset threshold, the candidate frame in the image to be detected is deleted, and the candidate frame in the segmented image is retained.
5. The target face image matching method according to claim 1, characterized in that: According to the remaining candidate frames and their associated facial key point information, the facial image portion pointed to by each candidate frame is aligned and cropped to obtain the facial image corresponding to each candidate frame, including the following steps: Determine whether the remaining candidate frames have corresponding facial key point information; When the candidate frame has corresponding facial key point information, determine the transformation matrix corresponding to the facial image from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the facial key point information; Align and crop the face image corresponding to the candidate frame from the to-be-detected picture or the segmented picture corresponding to the candidate frame according to the transformation matrix; When there is no corresponding facial key point information in the candidate frame, the size of the candidate frame is enlarged according to a preset amplitude, and a corresponding image is cropped from the to-be-detected picture or the segmented picture corresponding to the candidate frame as the facial image corresponding to the candidate frame.
6. The target face image matching method according to any one of claims 1 to 5, characterized in that: Searching a preset facial feature library according to the facial feature vector of the facial image to determine a preset facial feature vector similar to the facial feature vector, and outputting the search result, comprises the following steps: Calling a pre-trained image feature extraction model to extract a facial feature vector from the facial image; Performing similarity calculation on the facial feature vector and a plurality of preset facial feature vectors in a preset facial feature library to obtain a similarity calculation result; Arrange the similarities in the similarity calculation results in reverse order, and select some preset face feature vectors with the highest order as similar face feature vectors; Output the target face image matching result, which includes the target face image determined according to the uniqueness of the similar face feature vector.
7. A target face image matching device, characterized in that: include: A face detection module is used to perform face detection on the image to be detected and the segmented image corresponding to the human image therein, and obtain the candidate frame and face key point information corresponding to the face image part of the human image; The candidate optimization module is used to retain the candidate frames belonging to the segmented picture and delete the candidate frames belonging to the picture to be detected for the candidate frames corresponding to the same face image part, including: calculating the intersection-and-union score between the candidate frames of the picture to be detected and the segmented picture based on pixels; comparing the intersection-and-union score with a preset threshold; when the intersection-and-union score between the candidate frame of the picture to be detected and the candidate frame in the segmented picture is greater than the preset threshold, deleting the candidate frame in the picture to be detected and retaining the candidate frame in the segmented picture; A face alignment module is used to align and crop the face image portion pointed to by each candidate frame according to the remaining candidate frames and their associated face key point information to obtain the face image corresponding to each candidate frame; The similarity matching module is used to search a preset facial feature library according to the facial feature vector of the facial image to determine a preset facial feature vector similar to the facial feature vector, and output the search result.
8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.
