Key point positioning method, device and storage medium based on open set target detection

Abstract

The present application relates to a key point positioning method, device and storage medium based on open set target detection, the method comprising: obtaining a scene image, detecting all faces in the scene image through an open set target detection network, and obtaining a face image; dividing the face image into a high-quality face image and a low-quality face image through a face quality detection network, obtaining a final facial image with the highest similarity based on the first text information, predicting the facial key points of the final facial image through a pre-trained facial key point positioning network, and obtaining the first facial key point; extracting image features from the low-quality face image through a pre-trained image feature extraction network, and predicting the image features through a trained linear layer to obtain the second facial key point. The present application improves the accuracy and robustness of image key point positioning in complex scenes by fusing multimodal data such as images and texts.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a key point positioning method, device and storage medium based on open set target detection. Background Art

[0002] Face recognition technology plays a vital role in modern society and is widely used in security monitoring, identity authentication, human-computer interaction and other fields. As the core step of face recognition, the accuracy of face key point positioning directly affects the performance of subsequent recognition tasks. Traditional face key point positioning methods mainly rely on single modality data, such as face images or videos, and directly locate key points of the entire face image. These methods perform well under ideal conditions, but are prone to positioning errors in complex scenes. For example, posture changes may make it difficult to accurately detect some key points, and occlusion may cause key points to be missing. In addition, directly locating key points of the entire face image is computationally complex and difficult. Summary of the invention

[0003] The embodiments of the present application provide a key point positioning method, device and storage medium based on open set target detection, so as to at least solve the problem that the existing face recognition technology in the related art is prone to positioning errors in complex scenes, resulting in low recognition accuracy.

[0004] In a first aspect, an embodiment of the present application provides a key point positioning method based on open set target detection, including:

[0005] Acquire a scene image containing at least one human face, and detect all human faces in the scene image through an open set object detection network to obtain at least one human face image;

[0006] Performing quality detection on the face images through a face quality detection network, and dividing the face images into high-quality face images and low-quality face images;

[0007] Acquire first text information, and based on the first text information, acquire a final facial feature image with the highest similarity from the high-quality facial image, and predict facial key points of the final facial feature image using a pre-trained facial key point positioning network to obtain a first facial key point;

[0008] The image features of the poor-quality face image are extracted by a pre-trained image feature extraction network, and the image features are predicted by a trained linear layer to obtain second face key points; wherein the first face key points and the second face key points are used as positioning points of the scene image.

[0009] In one embodiment, obtaining the final facial features image with the highest similarity from the high-quality facial image includes:

[0010] Performing facial feature detection on the high-quality face image through the open set object detection network to obtain at least one set of facial feature images;

[0011] The similarity between the first text information and each group of facial feature images is calculated through a multimodal feature fusion network, and the group of facial feature images with the highest similarity is selected as the final facial feature images.

[0012] In one embodiment, the training process of the facial key point localization network includes:

[0013] Acquire high-quality image training data collected historically, and preprocess the high-quality image training data to meet the input requirements of the model;

[0014] Inputting the preprocessed high-quality image training data into the open set object detection network to perform facial features detection, thereby obtaining facial features image training data;

[0015] Acquire the first text information, perform image-text matching processing on the facial features image training data and the first text information through a multimodal feature fusion network, and acquire the facial features training image with the highest similarity;

[0016] Predicting facial key points of the facial feature training image using a facial key point positioning network to be trained to obtain first predicted key points;

[0017] Obtain a first real key point, calculate a first loss value according to the first predicted key point and the first real key point, and update the parameters of the face key point positioning network to be trained according to the first loss value until the first loss value meets the preset requirements, thereby obtaining the face key point positioning network.

[0018] In one embodiment, the training process of the image feature extraction network and the linear layer includes:

[0019] Acquire historically collected low-quality image training data, and preprocess the low-quality image training data to meet the input requirements of the model;

[0020] Perform feature extraction on the preprocessed low-quality image training data to obtain visual features and text features;

[0021] Comparing the visual features and the text features, calculating a contrast loss value, and fine-tuning the parameters of the image feature extraction network to be trained according to the contrast loss value until the contrast loss value meets a preset requirement, thereby obtaining the image feature extraction network;

[0022] Sending the inferior image training data to the image feature extraction network to obtain training visual features, predicting the training visual features through the linear layer to be trained to obtain a second prediction key point;

[0023] Obtain a second real key point, calculate a second loss value according to the second predicted key point and the second real key point, and update the parameters of the linear layer to be trained according to the second loss value until the second loss value meets the preset requirements, thereby obtaining a trained linear layer.

[0024] In one embodiment, the feature extraction of the preprocessed low-quality image training data to obtain visual features and text features includes:

[0025] Extracting features of the poor-quality image training data through the to-be-trained image feature extraction network to obtain the visual features;

[0026] The pre-processed low-quality image training data is converted into text data through a multimodal large model to obtain a detailed description text, and feature extraction is performed on the detailed description text through a text feature extraction network to obtain the text feature.

[0027] In one embodiment, comparing the visual features and the text features to calculate the contrast loss value includes:

[0028] Based on the comparison of visual features to text features, the image loss value is calculated;

[0029] Based on the comparison between text features and visual features, the text loss value is calculated;

[0030] The image loss value and the text loss value are weighted averaged to obtain the contrast loss value.

[0031] In one embodiment, detecting all faces in the scene image by an open set object detection network to obtain at least one face image includes:

[0032] Loading an open set target detection model, and setting weights, parameters, and configurations of the open set target detection model according to user requirements;

[0033] Acquire a prompt text for presetting, preprocess the scene image and the prompt text, and obtain data to be processed in a format suitable for the open set object detection model;

[0034] The open set target detection model processes the data to be processed and calculates the position and confidence of the face in the scene image;

[0035] Face extraction is performed in the scene image according to the position and confidence of the face to obtain at least one face image.

[0036] In one embodiment, performing quality detection on the face image through a face quality detection network to classify the face image into high-quality face images and low-quality face images includes:

[0037] Loading a face quality detection model, and preprocessing the face image to match the input requirement of the face quality detection model;

[0038] Performing forward propagation processing on the face image through the face quality detection model to obtain a quality score of the face image;

[0039] Obtain a quality score threshold preset by a user, compare the quality score with the quality score threshold, and when the quality score is higher than the quality score threshold, mark the face image as a high-quality face image; when the quality score is lower than or equal to the quality score threshold, mark the face image as a low-quality face image.

[0040] In a second aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the key point positioning method based on open set target detection as described in the first aspect above is implemented.

[0041] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the key point positioning method based on open set target detection as described in the first aspect above.

[0042] The key point positioning method, device and storage medium for open set target detection provided by the embodiments of the present application have at least the following technical effects:

[0043] This application divides the quality of an image containing multiple faces into good quality and poor quality face images, obtains text information, and then integrates multi-modal data such as image and text information to provide more comprehensive and accurate key point positioning results, overcome the limitations of single modality data, and solve the problem that existing face recognition technology is prone to positioning errors in complex scenes, resulting in low recognition accuracy.

[0044] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 is a flow chart of a key point positioning method for open set target detection in one embodiment of the present application;

[0047] Figure 2 is a flow chart of a face quality judgment process in one embodiment of the present application;

[0048] Figure 3 is a flow chart of a face key point location network training process in one embodiment of the present application;

[0049] Figure 4 is a flow chart of the image feature extraction network training and linear layer training process in one embodiment of the present application;

[0050] Figure 5 It is a structural block diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0053] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0054] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include unlisted steps or units, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0055] In order to solve the above problems, this application proposes a facial key point location method based on open set target detection. This method uses multimodal data fusion technology and comprehensively utilizes image and text information to provide more comprehensive and accurate key point location results. At the same time, a local key point location strategy is adopted to reduce the difficulty of direct regression and improve location accuracy and efficiency.

[0056] In one embodiment of the present application, a key point positioning method based on open set target detection is provided. The specific implementation process is referred to Figure 1 The implementation steps are described as follows.

[0057] Step S1, obtaining a scene image containing at least one face, detecting all faces in the scene image through an open set object detection network, and obtaining at least one face image.

[0058] Specific reference Figure 2, load the open set target detection network model, set the weights, parameters and configuration of the open set target detection model according to user needs; obtain the prompt text for pre-setting, such as the prompt text is "face", pre-process the scene image and the prompt text to obtain the data to be processed in a format suitable for the open set target detection model; the open set target detection model processes the data to be processed, calculates the position and confidence of the face in the scene image; extracts the face in the scene image according to the position and confidence of the face, and obtains at least one face image.

[0059] Open-set object detection refers to the ability of the model to recognize categories seen during training (known categories) and categories not seen during training (unknown categories) in the object detection task. That is, the open-set object detection network can recognize new categories that have not appeared in the training stage, which is the main difference from traditional closed-set object detection.

[0060] In this application, the open set target detection network used is Grounding DINO. By combining the Transformer-based detector DINO with true value pre-training, the detector can detect any object through human input (such as category names or referential expressions). The key solution for open set target detection is to introduce language into the closed set detector for open set concept generalization. In order to effectively integrate language and visual modalities, the closed set detector is divided into three stages, and a tight fusion solution is proposed, including a feature enhancer, a language-guided query selection and a cross-modal fusion, that is, Grounding DINO can detect specified targets based on text descriptions. For example, users can set the detection target of the open set target detection network according to their own needs. For example, if you input "detect faces in pictures", Grounding DINO can find face images from scene images.

[0061] Step S2, performing quality detection on the face image through a face quality detection network, and dividing the face image into high-quality face images and low-quality face images.

[0062] Specific reference Figure 2 , load a face quality detection model, pre-process the face image to match the input requirement of the face quality detection model; perform forward propagation processing on the face image through the face quality detection model to obtain a quality score of the face image, obtain a quality score threshold preset by a user, compare the quality score with the quality score threshold, and when the quality score is higher than the quality score threshold, mark the face image as a high-quality face image; when the quality score is lower than or equal to the quality score threshold, mark the face image as a low-quality face image.

[0063] The principle of the face quality detection algorithm used in this application is the same as that of face recognition. The image quality detection algorithm (FQ) is also based on the principle of feature extraction. It uses neural networks to learn from massive data to obtain the features that face quality detection focuses on (mainly including light, blur, angle, occlusion, expression, noise, etc.) and perform quality judgment.

[0064] exist Figure 2 In the example, the scene image (i.e., the original image) includes three faces, and the detection target is set to "face". Then, after detection by the open set target detection network, three face images are obtained. The three face images are then input into the face quality detection network, and the network makes a judgment on the quality of the face images. Before specific application, the user can first set the judgment conditions of the face quality detection network. For example, in the embodiment of the present application, the face image with no occlusion of the facial features can be set as a high-quality image, and the face image with any part of the facial features occluded can be identified as a low-quality image.

[0065] Step S3, obtaining first text information, and based on the first text information, obtaining a final facial feature image with the highest similarity from the high-quality facial image, and predicting facial key points of the final facial feature image through a pre-trained facial key point positioning network to obtain the first facial key point.

[0066] Specific reference Figure 3 , the step of obtaining the final facial features image with the highest similarity from the high-quality facial image comprises:

[0067] The high-quality face image is subjected to facial feature detection through the open set target detection network to obtain at least one group of facial feature images; the similarity between the first text information and each group of facial feature images is calculated through the multimodal feature fusion network, and the group of facial feature images with the highest similarity is selected as the final facial feature images.

[0068] In this embodiment, the training process of the facial key point positioning network includes: obtaining high-quality image training data collected historically, and preprocessing the high-quality image training data to adapt to the input requirements of the model; inputting the preprocessed high-quality image training data into the open set target detection network for facial feature detection to obtain facial feature image training data; obtaining first text information, performing image-text matching processing on the facial feature image training data and the first text information through a multimodal feature fusion network to obtain facial feature training images with the highest similarity; predicting facial key points on the facial feature training images through the facial key point positioning network to be trained to obtain first predicted key points; obtaining a first real key point, calculating a first loss value based on the first predicted key point and the first real key point, and updating the parameters of the facial key point positioning network to be trained based on the first loss value until the first loss value meets the preset requirements to obtain the facial key point positioning network.

[0069] The multimodal feature fusion network used in this application refers to a deep learning network that can process and integrate data from different modalities (such as vision, text, audio, etc.), and extract and fuse features from them for further analysis and decision-making. The core of this network is how to effectively combine information from different modalities to improve the performance of the task and enhance the understanding and generalization ability of the model.

[0070] The face key point localization network is a deep learning model specifically designed to detect and locate key points in a face. These key points usually include the locations of facial features such as eyes, nose, mouth, eyebrows, etc. Specifically, the process of automatically locating and identifying faces in images or videos, commonly used face detection methods are based on feature classifiers, such as Haar feature classifiers and deep learning models. These methods achieve face location and recognition by detecting the characteristic patterns and shapes of faces. Principle of face key point localization Face key point localization refers to the automatic location of key points of a face, such as eyes, nose, mouth, etc., in a face image. Commonly used key point localization methods are based on feature descriptors and regression models, such as manually designed features and deep learning models. These methods achieve the location and recognition of face key points by learning the location and shape patterns of face key points.

[0071] Specific as Figure 3 As shown in Figure 1, the user sets the search target of the open set object detection network to the facial features, and then Figure 2The good quality face image obtained in the open set target detection network is input into the open set target detection network to obtain the facial features image. In the process of facial features image detection, there will be false detection results, such as eyebrows being recognized as eyes, etc. Therefore, it is necessary to eliminate the images that do not meet the requirements through text description, so as to obtain the final facial features image that is most similar to the face in the high-quality image, and then input the final facial features image into the facial key point positioning network for positioning to obtain the predicted key points. In addition, in the model training stage, it is necessary to obtain the real key points ( Figure 3 The blue and red dots in the figure represent the key points of the facial features. The loss value is calculated based on the real key points and the predicted key points. If the loss value does not meet the model training requirements, the parameters of the face key point positioning network model to be trained are adjusted to carry out the next step of training.

[0072] Step S4, extracting image features from the inferior face image through a pre-trained image feature extraction network, and predicting the image features through a trained linear layer to obtain a second face key point; wherein the first face key point and the second face key point are used as positioning points of the scene image. Image feature extraction networks refer to a class of deep learning models whose main task is to automatically learn and extract useful feature representations from images. These features can capture key information in the image, such as shape, texture, color distribution, etc., which are crucial for tasks such as image recognition, classification, and detection.

[0073] Specific reference Figure 4 , the training process of the image feature extraction network and the linear layer includes:

[0074] Acquire historically collected low-quality image training data, and preprocess the low-quality image training data to adapt to the input requirements of the model; perform feature extraction on the preprocessed low-quality image training data to obtain visual features and text features. In this embodiment, the low-quality image training data is feature extracted by the image feature extraction network to be trained to obtain the visual features; the preprocessed low-quality image training data is converted into text data by a multimodal large model to obtain a detailed description text, and feature extraction is performed on the detailed description text by a text feature extraction network to obtain the text features.

[0075] Then, the visual features and text features are compared, the contrast loss value is calculated, and the parameters of the image feature extraction network to be trained are fine-tuned according to the contrast loss value, until the contrast loss value meets the preset requirements, the image feature extraction network is obtained; the inferior image training data is sent to the image feature extraction network to obtain the trained visual features, and the trained visual features are predicted by the linear layer to be trained to obtain the second predicted key point; the second real key point is obtained, the second loss value is calculated according to the second predicted key point and the second real key point, and the parameters of the linear layer to be trained are updated according to the second loss value, until the second loss value meets the preset requirements, the trained linear layer is obtained. In the present application, the requirements for the first loss value and the second loss value may be the same or different, and the specific training requirements may be set according to the actual needs of the user.

[0076] In this embodiment, the image loss value is calculated based on the comparison of visual features to text features; the text loss value is calculated based on the comparison of text features to visual features; the image loss value and the text loss value are weighted averaged to obtain the contrast loss value.

[0077] In this embodiment, the multimodal big model is an advanced model in the field of deep learning that can process and integrate information from multiple different data modalities. These modalities include but are not limited to text, images, videos, audio, etc. The goal of the multimodal big model is to understand and learn the associations between cross-modalities to improve the performance and generalization ability of the model on various tasks. The core of the multimodal big model is data fusion, that is, combining data from different modalities so that the model can understand information from multiple perspectives.

[0078] Figure 4 It is the training process of the image feature extraction network model and the linear layer. In the specific training process, first Figure 2 The training data, i.e., poor-quality facial images, are obtained by a method, and then these facial images are respectively input into the multimodal large model and the image feature extraction network to be trained to obtain initial visual features and detailed text features, and then the detailed text features are optimized through the text feature extraction network to obtain text features, and then the contrast loss value Contrastive Loss is calculated based on the text features and the initial visual features, and the parameters of the image feature extraction network model to be trained are adjusted by the contrast loss value. In this embodiment, the multimodal large model and the text feature extraction network use existing network modules, such as MiniCPM-V2.6, InternVL2, BERT, etc. The parameters of these models (such as weights) are fixed, while the model parameters of the image feature extraction network to be trained are not fixed and are obtained through training with a large amount of data.

[0079] Next, the poor quality face image is sent to the adjusted image feature extraction network model for feature extraction, and then the extracted image features are input into the linear layer to be trained. The linear layer predicts the key points of the image features, and then calculates the loss value based on the predicted key points and the real key points ( Figure 4 The blue and red dots on the face image in the figure represent key points). The parameters of the linear layer are adjusted according to the loss value, and the iterative training continues until the loss value meets the preset requirements.

[0080] To summarize, the present application divides the faces in an image containing multiple faces into good quality and poor quality face images by dividing the quality of the image, obtains text information, and then integrates multimodal data such as image and text information to provide more comprehensive and accurate key point positioning results, overcomes the limitations of single modality data, solves the problem that existing face recognition technology is prone to positioning errors in complex scenes, resulting in low recognition accuracy, and improves the accuracy and robustness of the face recognition system.

[0081] Specifically, in one embodiment, this embodiment describes in detail the key point positioning method based on open set target detection provided by the present application through specific formulas and algorithms.

[0082] Step 1: Face quality judgment process based on open set target detection

[0083] This process involves two networks, namely the open set object detection network and the face quality detection network. The specific process is as follows: first, any original image containing a face is sent to the open set object detection network Grounding DINO to detect all face images; secondly, the detected face images are sent to the face quality detection network FaceQnet to judge the quality of the face images; finally, the face images with good quality (referred to as data A) and the face images with poor quality (referred to as data B) are output.

[0084] Step 2: Face key point positioning network training process based on data A

[0085] This process involves three networks, namely the open set object detection network, the multimodal feature fusion network and the face key point localization network. Specifically, the high-quality face image obtained in step 1 is used as input and sent to the open set object detection network Grounding DINO to detect the facial features, including eyes, nose, and mouth (this process may have false detections).

[0086] In order to remove false detections and reconfirm the accuracy of facial features, this embodiment sends text information (including but not limited to text descriptions of eyes, nose, mouth, etc., which are input by the user) and detected facial features images to the multimodal feature fusion network BLIP, performs image-text matching, outputs the image-text similarity, selects the facial features image with the greatest similarity as the final image, and removes false detection images.

[0087] The facial features image is input into the facial key point location network, which uses a combination of Transformer architecture and linear layers to accurately predict the coordinates of the five key points: left eye, right eye, nose, left corner of mouth, and right corner of mouth. The facial key point network is trained by calculating the L1 Loss between the predicted key points and the actual key points. The loss function calculation formula used in this process is as follows:

[0088]

[0089] in, is L1 Loss; is the L1 norm; For facial key point localization network, is the facial image output by the multimodal feature fusion network. The predicted coordinates of the facial key points output by the facial key point positioning network. are the real coordinates of the key points of the facial features.

[0090] Step 3: Image feature extraction network training and linear layer training process based on data B:

[0091] This process involves two stages and two networks: the first stage is the fine-tuning training process of the image feature extraction network, including the image feature extraction network and the text feature extraction network; the second stage is the linear layer training process for the facial key point location task, including the image feature extraction network and the linear layer in stage 1. The specific process is as follows:

[0092] Phase 1: Using more detailed text descriptions of images can help the image feature extraction network learn better and more generalized image features. By using these detailed text descriptions to fine-tune the visual model, the feature extraction effect of poor quality face images can be significantly improved, making it pay more attention to details and fine-grained feature information, thereby achieving better results in subsequent key point regression tasks.

[0093] Therefore, here, the poor quality face image in step 1 is sent to a multimodal large model including but not limited to MiniCPM-V2.6 and InternVL2 to generate a detailed text description; then, the text feature extraction network BERT is used to extract the features of the detailed text, and the weights of the text feature extraction network are fixed. The image feature extraction network is fine-tuned by Contrastive Loss, which uses the Vision Transformer architecture. The Contrastive Loss used in this process includes two parts: image-to-text contrast loss and text-to-image contrast loss. The calculation formula of the total loss is:

[0094] The image-to-text contrast loss formula is as follows:

[0095]

[0096] The contrastive loss formula for text to image is as follows:

[0097]

[0098] The total contrast loss is:

[0099]

[0100] in, is the temperature parameter, is the total number of samples in the batch, Representing image features and text features The cosine similarity of . The formula is as follows:

[0101]

[0102] Stage 2: The poor quality face image in step 1 is sent to the image feature extraction network fine-tuned in stage 1. A linear layer is connected to the image feature extraction network to predict facial key points (left eye, right eye, nose, left corner of the mouth, right corner of the mouth). Finally, the linear layer is trained by calculating the L2 Loss between the predicted key points and the actual key points.

[0103] After the solution of this application is trained, it is applied to image processing. The specific application process is as follows:

[0104] For any image containing a face, all face images are detected through the open set object detection network Grounding DINO, and the face images are sent to the face quality detection network FaceQnet to output face images with good quality and face images with poor quality.

[0105] The good-quality face image is sent to the open-set object detection network Grounding DINO again to detect the facial features. Subsequently, the text information including but not limited to the text description of eyes, nose, mouth, etc. and the detected facial features are sent to the multimodal feature fusion network BLIP for image-text matching. By calculating the image-text similarity, the facial features image with the highest similarity is selected as the final image. Finally, the accurate facial features image is sent to the facial key point location network to predict the facial key points.

[0106] The poor quality face images are sent to the fine-tuned image feature extraction network to extract more generalized image features, and the extracted features are used to predict the face key points through the linear layer.

[0107] In summary, the key point positioning method based on open set target detection provided by this application solves the limitations of single modality data in complex scenes. This application provides more comprehensive and accurate key point positioning results by fusing multimodal data such as images and texts, overcoming the limitations of single modality data. In addition, the accuracy of key point positioning is also improved, that is, this application reduces the difficulty of direct regression and improves positioning accuracy and efficiency by adopting a local key point positioning strategy. In addition, this application can also enhance the robustness of the face recognition system, can effectively cope with complex scenes such as posture changes and occlusions, and provide more stable and reliable key point positioning results, thereby enhancing the robustness of the face recognition system.

[0108] In a second aspect, an embodiment of the present application provides an electronic device, Figure 5 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device may include a processor 11 and a memory 12 storing computer program instructions.

[0109] Specifically, the processor 11 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0110] Among them, the memory 12 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 12 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 12 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 12 may be inside or outside the data processing device. In a specific embodiment, the memory 12 is a non-volatile memory. In a specific embodiment, the memory 12 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0111] The memory 12 may be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 11 .

[0112] The processor 11 reads and executes computer program instructions stored in the memory 12 to implement any one of the key point positioning methods based on open set target detection in the above embodiments.

[0113] In one embodiment, the electronic device may further include a communication interface 13 and a bus 10. Figure 5 As shown, the processor 11, the memory 12, and the communication interface 13 are connected via a bus 10 and communicate with each other.

[0114] The communication interface 13 is used to realize the communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 13 can also realize data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage and image / data processing workstations.

[0115] The bus 10 includes hardware, software or both, and couples the components of the electronic device to each other. The bus 10 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 10 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 10 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0116] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the key point positioning method based on open set target detection provided in the first aspect is implemented.

[0117] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0118] In a possible implementation, the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the key point positioning method based on open set target detection provided in the first aspect.

[0119] Among them, the program code for executing the present application can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0120] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A key point localization method based on open set target detection, characterized in that: include: Acquire a scene image containing at least one human face, and detect all human faces in the scene image through an open set object detection network to obtain at least one human face image; Performing quality detection on the face images through a face quality detection network, and dividing the face images into high-quality face images and low-quality face images; Acquire first text information, perform image-text matching processing on the high-quality face image based on the first text information, obtain a final facial feature image with the highest similarity to the first text information from the high-quality face image, and predict facial key points of the final facial feature image using a pre-trained facial key point positioning network to obtain a first facial key point; The low-quality face image is subjected to feature extraction through a pre-trained image feature extraction network to obtain image features, and the image features are predicted through a trained linear layer to obtain second face key points; wherein the first face key points and the second face key points are used as positioning points of the scene image.

2. The key point positioning method according to claim 1, characterized in that: The step of obtaining a final facial feature image with the highest similarity from the high-quality facial image comprises: Performing facial feature detection on the high-quality face image through the open set object detection network to obtain at least one set of facial feature images; The similarity between the first text information and each group of facial feature images is calculated through a multimodal feature fusion network, and the group of facial feature images with the highest similarity is selected as the final facial feature images.

3. The key point positioning method according to claim 1, characterized in that: The training process of the facial key point positioning network includes: Acquire high-quality image training data collected historically, and preprocess the high-quality image training data to meet the input requirements of the model; Inputting the preprocessed high-quality image training data into the open set object detection network to perform facial features detection, thereby obtaining facial features image training data; Acquire the first text information, perform image-text matching processing on the facial features image training data and the first text information through a multimodal feature fusion network, and acquire the facial features training image with the highest similarity; Predicting facial key points of the facial features training image using a facial key point positioning network to be trained to obtain first predicted key points; Obtain a first real key point, calculate a first loss value according to the first predicted key point and the first real key point, and update the parameters of the face key point positioning network to be trained according to the first loss value until the first loss value meets the preset requirements, thereby obtaining the face key point positioning network.

4. The key point positioning method according to claim 1, characterized in that: The training process of the image feature extraction network and the linear layer includes: Acquire historically collected low-quality image training data, and preprocess the low-quality image training data to meet the input requirements of the model; Perform feature extraction on the preprocessed low-quality image training data to obtain visual features and text features; Comparing the visual features and the text features, calculating a contrast loss value, and fine-tuning the parameters of the image feature extraction network to be trained according to the contrast loss value until the contrast loss value meets a preset requirement, thereby obtaining the image feature extraction network; Sending the inferior image training data to the image feature extraction network to obtain training visual features, predicting the training visual features through the linear layer to be trained to obtain a second prediction key point; Obtain a second real key point, calculate a second loss value according to the second predicted key point and the second real key point, and update the parameters of the linear layer to be trained according to the second loss value until the second loss value meets the preset requirements, thereby obtaining a trained linear layer.

5. The key point positioning method according to claim 4, characterized in that: The feature extraction of the pre-processed inferior image training data to obtain visual features and text features includes: Extracting features from the poor-quality image training data using the image feature extraction network to be trained to obtain the visual features; The pre-processed low-quality image training data is converted into text data through a multimodal large model to obtain a detailed description text, and feature extraction is performed on the detailed description text through a text feature extraction network to obtain the text feature.

6. The key point positioning method according to claim 4, characterized in that: The comparing the visual features and the text features to calculate the contrast loss value includes: Based on the comparison of visual features to text features, the image loss value is calculated; Based on the comparison between text features and visual features, the text loss value is calculated; The image loss value and the text loss value are weighted averaged to obtain the contrast loss value.

7. The key point positioning method according to claim 1, characterized in that: The detecting all faces in the scene image by an open set object detection network to obtain at least one face image includes: Loading an open set target detection model, and setting weights, parameters, and configurations of the open set target detection model according to user requirements; Acquire a prompt text for presetting, preprocess the scene image and the prompt text, and obtain data to be processed in a format suitable for the open set object detection model; The open set target detection model processes the data to be processed and calculates the position and confidence of the face in the scene image; Face extraction is performed in the scene image according to the position and confidence of the face to obtain at least one face image.

8. The key point positioning method according to claim 1, characterized in that: Performing quality detection on the face image through a face quality detection network to divide the face image into high-quality face images and low-quality face images, including: Loading a face quality detection model, and preprocessing the face image to match the input requirement of the face quality detection model; Performing forward propagation processing on the face image through the face quality detection model to obtain a quality score of the face image; Obtain a quality score threshold preset by a user, compare the quality score with the quality score threshold, and when the quality score is higher than the quality score threshold, mark the face image as a high-quality face image; when the quality score is lower than or equal to the quality score threshold, mark the face image as a low-quality face image.

9. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the key point positioning method based on open set target detection as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the key point positioning method based on open set target detection according to any one of claims 1 to 8 is implemented.

Images