Training method, searching method, device and equipment of image character searching model

Abstract

The invention provides a training method, a search method, a device and equipment for an image character search model, which are applied to the technical field of artificial intelligence, and the training method comprises the following steps: firstly, constructing a data set comprising a base image face image, a target image and a sample label; then, inputting the base image and the target image into an image character search model, extracting feature maps under different scales, and performing feature comparison to generate a similarity feature map; and outputting a face matching result by integrating the similarity feature maps of different scales. And adjusting model parameters according to the face matching result of each group of data by utilizing a preset loss function, and finally obtaining a trained image character search model. According to the method, feature maps of the base map face image and the target image are extracted under multiple scales and are compared, so that detail features of the face on base maps of different scenes can be captured more effectively, non-compliant human portraits can be quickly detected and accurately recognized in a complex background, and compliance of media content is ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, a search method, a device and equipment for an image character search model. Background Art

[0002] With the rapid development of the Internet and digital media, a large amount of video and picture materials have been integrated on major online media platforms. These materials include a large number of portraits. Although most portraits are qualified, some non-compliant portraits are inevitable, which may have a significant negative impact on media platforms. For example, when an artist is listed as a bad artist due to improper behavior, the media platform needs to spend a lot of manpower and time to review and remove all materials containing the artist.

[0003] Therefore, how to achieve efficient and accurate image character search and ensure the compliance of media content has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of the above problems, the present invention provides a training method, a search method, a device and an apparatus for an image person search model that overcomes the above problems or at least partially solves the above problems. The technical solution is as follows:

[0005] A training method for an image person search model, comprising:

[0006] Obtaining a pre-constructed face image dataset, wherein the face image dataset includes at least one group of face image data, each group of face image data includes a base face image, a target image, and a sample label, and the sample label is used to indicate whether the target image contains face information in the base face image;

[0007] For any group of face image data in the face image data set: input the base face image and the target image of the face image data into an image person search model, extract face feature maps of the base face image at multiple different scales, and extract output feature maps of the target image at multiple different scales;

[0008] Using the face feature map and the output feature map to perform feature comparison, obtaining similarity feature maps between the base face image and the target image at multiple different scales;

[0009] Performing feature integration on the similarity feature maps at multiple different scales, and outputting a face matching result of the face image data;

[0010] The face matching results of each group of face image data are used to adjust the model parameters of the image person search model based on a preset loss function to obtain the trained image person search model.

[0011] Optionally, the using the face feature map and the output feature map to perform feature comparison to obtain similarity feature maps between the base face image and the target image at multiple different scales includes:

[0012] The facial feature maps at multiple different scales are respectively used as convolution kernels, and then convolution operations are performed with the output feature maps at multiple different scales to obtain similarity feature maps between the base face image and the target image at multiple different scales.

[0013] Optionally, the step of integrating the similarity feature maps at multiple different scales and outputting a face matching result of the face image data includes:

[0014] Downsampling and channel-connecting the similarity feature maps at multiple different scales to obtain a feature information integration feature map;

[0015] The feature information is integrated with the feature graph for full connection, and the face matching result of the face image data is output.

[0016] Optionally, extracting facial feature maps of the base face image at multiple scales includes:

[0017] Extracting a first facial feature map of the base facial image at a first preset scale using a first pre-trained feature extraction network;

[0018] Downsampling the first facial feature map to obtain a second facial feature map of the base facial image at a second preset scale;

[0019] The second facial feature map is downsampled to obtain a third facial feature map of the base facial image at a third preset scale.

[0020] Optionally, the first preset scale is n×c×7×7, wherein n is the number of training samples used in each parameter update, and c is the number of channels of the feature map.

[0021] Optionally, extracting output feature maps of the target image at multiple scales includes:

[0022] Extracting a first effective feature map of the target image at a fourth preset scale using a second pre-trained feature extraction network;

[0023] Downsampling and rotating the first effective feature map to capture features, to obtain a second effective feature map of the target image at a fifth preset scale;

[0024] Downsampling and rotating the second effective feature map to capture features, to obtain a third effective feature map of the target image at a sixth preset scale;

[0025] Using the third effective feature map as a third output feature map, upsampling the third output feature map and performing feature fusion with the second effective feature map to obtain a second output feature map;

[0026] Upsampling the third output feature map and performing feature fusion with the second effective feature map to obtain a second output feature map;

[0027] The second output feature map is upsampled and then feature fused with the first effective feature map to obtain a first output feature map.

[0028] A method for searching for people in an image, comprising:

[0029] Obtain video data and a base face image of a person to be searched;

[0030] Splitting the video data into single-frame video images;

[0031] For any of the single-frame video images: inputting the single-frame video image and the base face image into any of the above-mentioned image person search models to obtain a face matching result output by the image person search model;

[0032] Based on the face matching results of each of the single-frame video images, a person search result of the video data is generated.

[0033] A training device for an image person search model comprises: a face image data set acquisition unit, a feature extraction unit, a feature comparison unit, a feature integration unit and a model parameter adjustment unit.

[0034] The face image data set obtaining unit is used to obtain a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains face information in the base face image;

[0035] The feature extraction unit is used for: for any group of the face image data in the face image data set: inputting the base face image and the target image of the face image data into the image person search model, extracting face feature maps of the base face image at multiple different scales, and extracting output feature maps of the target image at multiple different scales;

[0036] The feature comparison unit is used to perform feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales;

[0037] The feature integration unit is used to perform feature integration on the similarity feature maps at multiple different scales and output a face matching result of the face image data;

[0038] The model parameter adjustment unit is used to use the face matching results of each group of face image data to adjust the model parameters of the image person search model based on a preset loss function to obtain the trained image person search model.

[0039] An image person search device comprises: a person data acquisition unit, a video splitting unit, a face matching result acquisition unit and a person search result acquisition unit,

[0040] The character data obtaining unit is used to obtain the video data and the base face image of the character to be searched;

[0041] The video splitting unit is used to split the video data into single-frame video images;

[0042] The face matching result obtaining unit is used for: for any of the single-frame video images: inputting the single-frame video image and the base face image into the above-mentioned image person search model to obtain the face matching result output by the image person search model;

[0043] The person search result obtaining unit is used to generate a person search result of the video data based on the face matching result of each of the single-frame video images.

[0044] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute any of the above-mentioned image person search model training methods, and / or to execute the above-mentioned image person search method.

[0045] By means of the above technical scheme, the training method, search method, device and equipment of the image person search model provided by the present invention obtain a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains the face information in the base face image; for any group of face image data in the face image data set: the base face image and the target image of the face image data are input into the image person search model, and the face feature map of the base face image at multiple different scales is extracted, and the output feature map of the target image at multiple different scales is extracted; the face feature map and the output feature map are used to perform feature comparison to obtain the similarity feature map between the base face image and the target image at multiple different scales; the similarity feature maps at multiple different scales are feature integrated to output the face matching results of the face image data; the model parameters of the image person search model are adjusted based on the preset loss function using the face matching results of each group of face image data to obtain a trained image person search model. By extracting and comparing feature maps from base face images and target images at multiple scales, the present invention can more effectively capture the detailed features of faces on base maps of different scenes, thereby quickly detecting and accurately identifying non-compliant portraits in complex backgrounds and ensuring the compliance of media content.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 A schematic flow chart showing an implementation of a method for training an image person search model provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a model structure of an image character search model provided by an embodiment of the present invention is shown;

[0050] Figure 3 A structural schematic diagram of a DEM module provided by an embodiment of the present invention is shown;

[0051] Figure 4A schematic diagram of the principle process of rotational convolution provided by an embodiment of the present invention is shown;

[0052] Figure 5 A schematic diagram of the structure of a feature comparison module provided by an embodiment of the present invention is shown;

[0053] Figure 6 A schematic diagram of the structure of a feature integration module provided by an embodiment of the present invention is shown;

[0054] Figure 7 A schematic diagram showing a flow chart of an implementation of a method for searching for people in an image provided by an embodiment of the present invention;

[0055] Figure 8 A schematic diagram showing the structure of a training device for an image person search model provided by an embodiment of the present invention is shown;

[0056] Fig. 9 A schematic diagram showing the structure of an image character search device provided by an embodiment of the present invention is shown;

[0057] Fig.10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0059] At present, most online media platforms mainly rely on manual review to handle media materials containing non-compliant portraits, but this method has many shortcomings. Especially in some special cases, such as when the portrait is rotated and embedded in the image, or the portrait is small in size, these will greatly increase the difficulty and workload of the reviewer. Therefore, designing a search method and process that can quickly and accurately detect specified people in video and image materials is of great significance to reducing the workload of reviewers and improving review efficiency.

[0060] In addition to manual review, media platforms also rely on the following two technical solutions when reviewing portraits of people in video and image materials:

[0061] 1. Feature-based face recognition method: First, perform face detection in the image to be detected, and then align the detected faces. Next, extract features from the aligned faces and the faces in the base image. Finally, calculate the feature similarity between the aligned faces and the base image faces. If the similarity is greater than a preset threshold, they are judged to be the same person. However, this method has limited effect when facing complex scenes. In particular, when the face is rotated, the scale changes, or the background is complex, the accuracy of feature extraction and matching will drop significantly.

[0062] 2. Distance-based face comparison method: It is mainly used to search for similar faces in the face database and match them by calculating image similarity. Although it is fast, the search has great limitations and poor accuracy, making it difficult to cope with diverse and complex practical application scenarios.

[0063] In view of the above problems, the present invention firstly considers automatically retrieving portraits of people in videos and images through algorithms to improve the review efficiency and reduce the labor intensity of reviewers.

[0064] However, when implementing person detection in a target image based on the target face base map, the following technical difficulties are faced:

[0065] Scale inconsistency: The portraits of people in the target image may be of different sizes, requiring the ability to handle different scales.

[0066] Anti-rotation capability: The portraits of people in the target image may be rotated at different angles, so the method is required to have anti-rotation capability.

[0067] Open set classification capability: The portrait of a person in the target image is usually different from the base image. The base image may be a passport photo, while the target image may be a photo of daily life, a still photo, or an artistic photo. The differences in background, posture, and environment require the method to have strong open set classification capabilities, that is, strong face recognition capabilities.

[0068] Based on this, in an embodiment of the present invention, a training method for an image character search model is provided. First, a data set including a base face image, a target image and sample labels is constructed. Then, the base image and the target image are input into the image character search model, feature maps at different scales are extracted, and feature comparison is performed to generate a similarity feature map. By integrating the similarity feature maps of different scales, the face matching result is output. Using a preset loss function, the model parameters are adjusted according to the face matching results of each group of data, and finally a trained image character search model is obtained. The present invention not only improves the accuracy and robustness of character recognition through multi-scale feature extraction and comparison, feature integration and model optimization, but also significantly improves the review efficiency, reduces the misjudgment rate, and ensures the compliance of media content.

[0069] like Figure 1 As shown, a flow chart of an implementation of a method for training an image person search model provided by an embodiment of the present invention is provided. The training method may include:

[0070] S100. Obtain a pre-constructed face image dataset, wherein the face image dataset includes at least one group of face image data, each group of face image data includes a base face image, a target image, and a sample label, and the sample label is used to indicate whether the target image contains face information in the base face image.

[0071] The base face image refers to a face image used as a reference standard. The base face image is clear and standardized, usually with a simple background, standard posture, uniform lighting, and clear facial features, and can be used to compare and match faces in other images.

[0072] Optionally, in order to pay more attention to facial features, the base face image can be an aligned face image. Specifically, the embodiment of the present invention can perform face detection on the base face image to obtain facial key points, and then input the base image, facial key points and standard face key points into an affine transformation matrix to obtain the base face image after face alignment.

[0073] Among them, the target image refers to the image that needs to be subjected to face detection and face recognition. In the embodiment of the present invention, the main task of image face search is to determine whether the target image contains the same face as the base face image. The target image may contain multiple faces and / or complex background information. The target image may include various types of photos, such as: life photos, stills, artistic photos, etc. These photos are usually taken in various environments, and may have different backgrounds, postures and lighting conditions, which may cause certain recognition difficulties.

[0074] The format of face image dataset can be dataset={[face i ,img i ,label i ], i = 1 ~ n}, indicating that the face image dataset contains n groups of face image data, each group of face image data consists of the following three parts: face i : base face image, img i : target image and label i : Sample label. When the sample label is "1", it means that the corresponding face image data is a positive sample, that is, the target image in the face image data contains the face information in the base face image. When the sample label is "1", it means that the corresponding face image data is a negative sample, that is, the target image in the face image data does not contain the face information in the base face image.

[0075] In an embodiment of the present invention, constructing a face image dataset may include the following two methods: one is to collect face image data to construct the dataset, and the other is to construct the dataset based on an existing face recognition dataset.

[0076] In method 1, the embodiment of the present invention can collect K image data of N people (including faces) according to the requirements of face recognition. Store the data of the same person in the same folder. Traverse all folders: extract the aligned faces of all images according to the face alignment method. Obtain M aligned face images and M original images. Randomly select one from each of the M aligned face images and the M original images to form a group of data. According to this process, the M original images can form at most M×M groups of data to form positive sample data. Randomly select a face data and an original image that is not this person or other background images that do not contain people to form a group of negative samples. Repeat K times to obtain K groups of negative sample data. Add the positive sample data and the negative sample data to the data set to form a face image data set. In actual operation, the embodiment of the present invention can timely collect the image data of the target person and store it in a separate folder, and execute the above process to construct a face image data set to improve the recognition effect of the model on the target person.

[0077] In the second method, the embodiment of the present invention can obtain the original data of an open source face recognition dataset, such as WebFace, WildFace and other open source datasets. Store the data of the same person in the same folder. Traverse all folders: extract the aligned faces of all images according to the face alignment method. Obtain M aligned face images and M original images. Randomly select one from each of the M aligned face images and the M original images to form a group of data. According to this process, the M original images can form at most M×M groups of data to form positive sample data. Randomly select a face data and an original image that is not the person or other background images that do not contain people to form a group of negative samples. Repeat K times to obtain K groups of negative sample data. Add the positive sample data and the negative sample data to the data set to form a model training data set. In actual operation, the embodiment of the present invention can store the image data of the target person collected in advance in a separate folder, and execute the above process to construct a face image data set to improve the recognition effect of the model on the target person.

[0078] S110. For any group of face image data in the face image data set: input the base face image and the target image of the face image data into the image person search model, extract the face feature map of the base face image at multiple different scales, and extract the output feature map of the target image at multiple different scales.

[0079] Among them, the face feature map is a feature representation extracted from the base face image at different scales, which contains multi-dimensional features about the face, such as facial features and skin texture.

[0080] Among them, the output feature map is a feature representation extracted from the target image at different scales, which contains the features of all potential faces in the target image and is used to compare with the features of the base face image.

[0081] The image person search model is a deep learning convolutional neural network model used to search for specific faces in the target image. The image person search model extracts features and compares them by inputting the base face image and the target image to determine whether the target image contains the face information in the base face image.

[0082] Optionally, the model structure of the image person search model can be as follows Figure 2 As shown in the figure, "backbone" in deep learning refers to a pre-trained convolutional neural network (such as ResNet, VGG, Inception, etc.) used to extract high-level features of the image. The FM (Feature Map) module represents the feature layer extracted from the image, and the number behind it represents the scale of the feature map corresponding to the feature layer, which is also the feature representation of different receptive fields. For example, the feature map scale corresponding to the "FM7" feature layer can be n×c×7×7, which means that the receptive field is large and can capture more global information. The feature map scale corresponding to the "FM5" feature layer can be n×c×5×5, which means that the receptive field is medium and can capture medium-scale information. The feature map scale corresponding to the "FM3" feature layer can be n×c×3×3, which means that the receptive field is small and can capture more local information. Among them, n is the number of training samples used in each parameter update, and c is the number of channels of the feature map.

[0083] The EFM (Effective Feature Map) module represents the effective feature map extracted from the input image (img) during the feature extraction process.

[0084] The OFM (Output Feature Map) module represents the output feature map after the feature extraction and processing process.

[0085] The DEM (Deep Enhancement Model) module represents a model that improves image processing effects through deep learning technology. The DEM module provided by the embodiment of the present invention combines the calculation principle of conventional convolution and image plane rotation to learn the angle information of the image while performing feature extraction.

[0086] The structure of the DEM module provided in the embodiment of the present invention can be as follows: Figure 3 As shown, the DEM module includes downsampling (downsample) and rotation convolution (Rconv2d).

[0087] The principle process of rotation convolution is as follows Figure 4 As shown in the figure, first, the outer product (coordinateKernel) of the position matrix of the conventional convolution conv2d and the input map (inputMap) is used as the convolution kernel, and then the convolution kernel is used to perform a convolution operation with the input map to obtain the angle weight map (angleMap), and then a rotation convolution kernel (rotateKernel) is generated based on the angle weight of the angle weight map, and finally the rotation convolution kernel is used to perform a convolution operation with the input map to obtain the output map (outputMap).

[0088] The DEM module provided in the embodiment of the present invention innovatively combines the calculation principles of conventional convolution and image plane rotation. This combination can not only obtain richer image information during feature extraction, but also pay special attention to the angle information of the image, thereby improving the matching ability of the FCM module through the angle information.

[0089] FCM (Feature Comparison Module) is used to calculate the similarity between the features of the base face image (face) and the target image (img) and generate multiple output feature layers.

[0090] FMM (Feature Merging Module) is used to integrate these output feature layers and generate the final matching results through downsampling, channel splicing and full connection operations.

[0091] S120, performing feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales.

[0092] Specifically, the embodiment of the present invention can perform convolution operations at different scales through a sliding window and anchor mechanism to calculate the degree of matching between the face feature map and the output feature map. Furthermore, the embodiment of the present invention can use different face feature maps of the base face image as convolution kernels and perform convolution operations with the feature layer of the target image to obtain the similarity score of each position, thereby capturing the feature similarity information at different scales and obtaining the similarity feature maps of the two at multiple different scales.

[0093] like Figure 2The model structure shown in the figure is that the FM module between the feature comparison module and the feature integration module is a data processing and matching layer, which does not contain learning parameters. Its main significance includes: the feature matrix extracted by convolution can be regarded as the vector representation of the image. In face recognition, the usual method is to embed two faces and then calculate their similarity. Therefore, the face feature map is used as a convolution kernel to perform a convolution operation with the output feature map, which is equivalent to calculating the feature distance between the vector of the face feature map and the local area (i.e., anchor point) of the output feature map in sequence according to a certain step size.

[0094] S130, integrating features of a plurality of similarity feature maps at different scales, and outputting a face matching result of the face image data.

[0095] Specifically, the embodiment of the present invention can integrate feature similarity information from different scales and different positions, process the integrated features through a fully connected layer, and finally output a face matching result (i.e., whether the target image contains the face information in the base face image).

[0096] S140, using the face matching results of each group of face image data, adjusting the model parameters of the image person search model based on a preset loss function to obtain a trained image person search model.

[0097] Specifically, the embodiments of the present invention can evaluate the difference between the face matching result of the face image data and the corresponding sample label based on a preset loss function, and adjust the model parameters of the image person search model by minimizing the value of the preset loss function to optimize the image person search model, so that the face matching result output by the optimized image person search model is as close to the corresponding sample label as possible. Usually a target threshold is set, and when the value of the preset loss function of the image person search model is less than the target threshold, it is confirmed that the image person search model training is completed. It can be understood that the specific value of the target threshold can be set according to the actual model prediction accuracy requirements.

[0098] Optionally, the preset loss function may be one of mean square error (MSE), cross-entropy loss (Cross-Entropy Loss) and mean absolute error (MAE).

[0099] The present invention provides a training method for an image person search model, the method comprising: obtaining a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains the face information in the base face image; for any group of face image data in the face image data set: inputting the base face image and the target image of the face image data into the image person search model, extracting face feature maps of the base face image at multiple different scales, and extracting output feature maps of the target image at multiple different scales; performing feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales; performing feature integration on the similarity feature maps at multiple different scales, and outputting face matching results of the face image data; using the face matching results of each group of face image data, adjusting the model parameters of the image person search model based on a preset loss function to obtain a trained image person search model. By extracting and comparing feature maps from base face images and target images at multiple scales, the present invention can more effectively capture the detailed features of faces on base maps of different scenes, thereby quickly detecting and accurately identifying non-compliant portraits in complex backgrounds and ensuring the compliance of media content.

[0100] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the present invention, the face feature map and the output feature map are used for feature comparison to obtain similarity feature maps between the base face image and the target image at multiple different scales, which may specifically include:

[0101] The facial feature maps at multiple scales are used as convolution kernels, and then convolved with the output feature maps at multiple scales to obtain similarity feature maps between the base face image and the target image at multiple scales.

[0102] Specifically, the embodiment of the present invention can use the feature comparison module of the image character search model to extract the similarity features of the output feature maps of different scales of the face feature maps at different scales through the sliding window and anchor point mechanism. Figure 2 The model structure shown in Figure 2, the structure of the feature comparison module can be as follows: Figure 5 As shown, in an embodiment of the present invention, the three feature layers FM7, FM5 and FM3 of the base face image can be used as convolution kernels respectively, and convolution operations are performed with the three feature layers OFM1, OFM2 and OFM3 of the target image to obtain 9 output layers OM1 to OM9, each output layer corresponding to a feature match, thereby calculating the feature similarity between the base face image and the target image at different scales.

[0103] The embodiment of the present invention uses facial feature maps of different scales as convolution kernels, which can capture features of different levels and details in the image, thereby effectively extracting a variety of information ranging from local details to overall structures, and then performing convolution operations in combination with output feature maps of different scales. This allows the model to effectively perform face matching in the face of various scale changes and posture changes, and can especially enhance the model's robustness to image scaling, rotation, translation and other transformations, thereby improving the model's adaptability in real scenarios and the reliability of face matching results.

[0104] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the present invention, the similarity feature maps at multiple different scales are integrated to output the face matching result of the face image data, which may specifically include:

[0105] The similarity feature maps at multiple different scales are downsampled and channel-connected to obtain a feature information integration feature map; the feature information integration feature map is fully connected to output the face matching result of the face image data.

[0106] Among them, channel connection (contact) refers to the operation of splicing multiple feature maps in the channel dimension. Channel connection can integrate feature maps from different scales, so that the model can simultaneously use the information of feature maps at multiple different scales. For example, if the size of each feature map is H×W×C, then the size of the spliced ​​feature map is H×W×(9C).

[0107] Among them, the fully connected (Linear) layer, also known as the fully connected layer, is a neural network layer used to convert the feature map into a classification result, that is, a face matching result, through a linear transformation and an activation function.

[0108] Specifically, the embodiment of the present invention can use the feature integration module of the image character search model to perform feature integration on the output layer of the feature comparison module to output the face matching result of the face image data. Figure 2 The model structure shown in Figure 2, the structure of the feature integration module can be as follows: Figure 6 As shown, the embodiment of the present invention can downsample the 9 output layers OM1 to OM9 to the same size, and then perform channel stitching to stitch the downsampled feature layers in the channel dimension to form a larger feature map, and then perform a full connection operation on the spliced ​​feature map to finally output the face matching result.

[0109] The embodiment of the present invention can integrate information of different scales into one feature map by downsampling and channel connecting similarity feature maps of different scales, thereby effectively combining local details and global structures, improving the comprehensive ability of feature representation, and making the model more accurate when searching for people in images. Through the full connection operation, the feature information is further integrated and optimized, making the final face matching result more accurate.

[0110] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the present invention, extracting facial feature maps of the base face image at multiple different scales may specifically include:

[0111] A first pre-trained feature extraction network is used to extract a first face feature map of a base face image at a first preset scale; the first face feature map is down-sampled to obtain a second face feature map of the base face image at a second preset scale; and the second face feature map is down-sampled to obtain a third face feature map of the base face image at a third preset scale.

[0112] Optionally, the first preset scale is n×c×7×7, where n is the number of training samples used in each parameter update, and c is the number of channels of the feature map. The second preset scale may be n×c×5×5, and the third preset scale may be n×c×3×3.

[0113] based on Figure 2 In the model structure shown, the main function of the downsampling convolution layer (Downsamp conv2d) is to obtain feature maps of different scales by reducing the resolution of the feature map, thereby increasing the receptive field, so that the model network can extract more contextual information at a lower resolution, which is more conducive to the comprehensive expression of facial image features. In the embodiment of the present invention, a first facial feature map with a scale of n×c×7×7 can be output from the first pre-trained feature extraction network "backboneB", a second facial feature map with a scale of n×c×5×5 can be obtained through a downsampling convolution layer, and a third facial feature map with a scale of n×c×3×3 can be obtained through a downsampling convolution layer. The first facial feature map, the second facial feature map and the third facial feature map are input into the feature comparison module.

[0114] exist Figure 2 In the model structure shown, "backboneB" has three feature layers. In actual applications, it can also be four layers, for example, the scales of the feature layers are n×c×9×9, n×c×7×7, n×c×5×5, and n×c×3×3. It can be understood that the number of feature layers corresponding to "backboneB" can be set according to actual needs. The larger the number, the more complex the model is, and the higher the requirements for computer resources.

[0115] It should be noted that the size of the feature layer corresponding to "backboneB" is usually an odd number to prevent the model from learning to lose its central properties.

[0116] The embodiment of the present invention can effectively obtain multi-scale features by extracting facial feature maps at different preset scales and performing downsampling step by step. These features cover different information from local details to global structures, which helps to improve the accuracy of face recognition.

[0117] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the present invention, extracting output feature maps of the target image at multiple different scales may specifically include:

[0118] A first effective feature map of the target image at a fourth preset scale is extracted using a second pre-trained feature extraction network; the first effective feature map is downsampled and rotated to capture features to obtain a second effective feature map of the target image at a fifth preset scale; the second effective feature map is downsampled and rotated to capture features to obtain a third effective feature map of the target image at a sixth preset scale; the third effective feature map is used as a third output feature map, and the third output feature map is upsampled and then feature fused with the second effective feature map to obtain a second output feature map; the third output feature map is upsampled and then feature fused with the second effective feature map to obtain a second output feature map; the second output feature map is upsampled and then feature fused with the first effective feature map to obtain a first output feature map.

[0119] It can be understood that the fourth preset scale, the fifth preset scale and the sixth preset scale can be set according to requirements.

[0120] based on Figure 2According to the model structure shown, the embodiment of the present invention can output the first effective feature map of the fourth preset scale from the first pre-trained feature extraction network "backboneA". The first effective feature map is input into the DEM module for downsampling and rotation feature capture to obtain the second effective feature map of the fifth preset scale. The second effective feature map is input into the DEM module for downsampling and rotation feature capture to obtain the third effective feature map of the sixth preset scale. Then, the third effective feature map is used as the third output feature map, and the third output feature map is upsampled to the fifth preset scale and then feature fused with the second effective feature map to obtain the second output feature map of the fifth preset scale. Then, the second output feature map is upsampled to the fourth preset scale and then feature fused with the first effective feature map to obtain the first output feature map of the fourth preset scale. The first output feature map, the second output feature map and the third output feature map are input into the feature comparison module.

[0121] It is understandable that the network structures of "backboneA" and "backboneB" can be the same or different. It should be noted that the network parameters of "backboneA" and "backboneB" are different. The embodiment of the present invention can use the model weights that have been trained in other face recognition tasks to initialize "backboneA" and "backboneB", and perform fine tuning on this basis.

[0122] The embodiments of the present invention use a pre-trained feature extraction network to extract target image features at different preset scales, and perform downsampling, rotation feature capture, upsampling and feature fusion step by step, so as to effectively acquire and integrate multi-scale features, enhance feature expression capabilities, and thus improve the model's image face recognition and matching capabilities.

[0123] like Figure 7 As shown, a flowchart of an implementation of an image person search method provided by an embodiment of the present invention is provided. The method may include:

[0124] S700: Obtain video data and a base face image of a person to be searched.

[0125] S710: Split the video data into single-frame video images.

[0126] S720. For any single-frame video image: input the single-frame video image and the base face image into the image person search model of the above-mentioned image person search model training method to obtain the face matching result output by the image person search model.

[0127] S730: Generate a person search result of the video data based on the face matching results of each single-frame video image.

[0128] The image character search method provided by the embodiment of the present invention splits the video data into single-frame images, uses the image character search model to perform accurate face matching on each frame, and then combines the matching results of all frames to generate the final character search results, thereby quickly and accurately searching for video frames with non-compliant character portraits in the video, thereby improving the processing efficiency of the media platform for related video frames and ensuring the compliance of the media content.

[0129] Although operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0130] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0131] Corresponding to the above-mentioned training method for image person search model, the embodiment of the present invention also provides a training device for image person search model, the structure of which is as follows: Figure 8 As shown, it may include: a face image data set obtaining unit 10, a feature extraction unit 11, a feature comparison unit 12, a feature integration unit 13 and a model parameter adjustment unit 14.

[0132] The face image data set acquisition unit 10 is used to obtain a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains the face information in the base face image.

[0133] The feature extraction unit 11 is used for any group of face image data in the face image data set: inputting the base face image and the target image of the face image data into the image character search model, extracting the face feature map of the base face image at multiple different scales, and extracting the output feature map of the target image at multiple different scales.

[0134] The feature comparison unit 12 is used to perform feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales.

[0135] The feature integration unit 13 is used to perform feature integration on a plurality of similarity feature maps at different scales and output a face matching result of the face image data.

[0136] The model parameter adjustment unit 14 is used to use the face matching results of each group of face image data to adjust the model parameters of the image person search model based on a preset loss function to obtain a trained image person search model.

[0137] Optionally, the feature comparison unit 12 can be specifically used to use facial feature maps at multiple scales as convolution kernels, and then perform convolution operations with output feature maps at multiple scales, so as to obtain similarity feature maps between the base face image and the target image at multiple scales.

[0138] Optionally, the feature integration unit 13 can be specifically used to downsample and channel-connect similarity feature maps at multiple scales to obtain a feature information integration feature map; fully connect the feature information integration feature map to output the face matching result of the face image data.

[0139] Optionally, the feature extraction unit 11 may include: a face feature map extraction subunit.

[0140] The facial feature map extraction subunit is used to use a first pre-trained feature extraction network to extract a first facial feature map of a base face image at a first preset scale; downsample the first facial feature map to obtain a second facial feature map of the base face image at a second preset scale; downsample the second facial feature map to obtain a third facial feature map of the base face image at a third preset scale.

[0141] Optionally, the first preset scale is n×c×7×7, where n is the number of training samples used in each parameter update, and c is the number of channels of the feature map.

[0142] Optionally, the feature extraction unit 11 may include: an output feature map extraction subunit.

[0143] The output feature map extraction subunit is used to extract a first effective feature map of the target image at a fourth preset scale using a second pre-trained feature extraction network; downsample and rotate the first effective feature map to capture features, and obtain a second effective feature map of the target image at a fifth preset scale; downsample and rotate the second effective feature map to capture features, and obtain a third effective feature map of the target image at a sixth preset scale; use the third effective feature map as the third output feature map, and upsample the third output feature map and perform feature fusion with the second effective feature map to obtain a second output feature map; upsample the third output feature map and perform feature fusion with the second effective feature map to obtain a second output feature map; upsample the second output feature map and perform feature fusion with the first effective feature map to obtain a first output feature map.

[0144] The present invention provides a training device for an image person search model, which is used to: obtain a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains the face information in the base face image; for any group of face image data in the face image data set: input the base face image and the target image of the face image data into the image person search model, extract the face feature map of the base face image at multiple different scales, and extract the output feature map of the target image at multiple different scales; use the face feature map and the output feature map to perform feature comparison to obtain the similarity feature map of the base face image and the target image at multiple different scales; perform feature integration on the similarity feature maps at multiple different scales, and output the face matching results of the face image data; use the face matching results of each group of face image data to adjust the model parameters of the image person search model based on a preset loss function to obtain a trained image person search model. By extracting and comparing feature maps from base face images and target images at multiple scales, the present invention can more effectively capture the detailed features of faces on base maps of different scenes, thereby quickly detecting and accurately identifying non-compliant portraits in complex backgrounds and ensuring the compliance of media content.

[0145] Corresponding to the above-mentioned image person search method, the embodiment of the present invention also provides an image person search device, whose structure is as follows: Fig. 9 As shown, it includes: a character data obtaining unit 20, a video splitting unit 21, a face matching result obtaining unit 22 and a character search result obtaining unit 23.

[0146] The character data obtaining unit 20 is used to obtain the video data and the base face image of the character to be searched.

[0147] The video splitting unit 21 is used to split the video data into single-frame video images.

[0148] The face matching result obtaining unit 22 is used for any single-frame video image: inputting the single-frame video image and the base face image into the above-mentioned image person search model to obtain the face matching result output by the image person search model.

[0149] The person search result obtaining unit 23 is used to generate a person search result of the video data based on the face matching results of each single frame of video image.

[0150] The image character search device provided by the embodiment of the present invention splits the video data into single-frame images, uses the image character search model to perform accurate face matching on each frame, and then generates the final character search result by integrating the matching results of all frames, so as to quickly and accurately search for video frames with non-compliant character portraits in the video, thereby improving the processing efficiency of the media platform for related video frames and ensuring the compliance of the media content.

[0151] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0152] The training device of the image character search model includes a processor and a memory. The above-mentioned face image data set acquisition unit 10, feature extraction unit 11, feature comparison unit 12, feature integration unit 13 and model parameter adjustment unit 14 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to extract and compare feature maps of the base face image and the target image at multiple scales, so as to more effectively capture the detailed features of the face on the base map of different scenes, so as to quickly detect and accurately identify non-compliant portraits in complex backgrounds and ensure the compliance of media content.

[0153] The image character search device includes a processor and a memory. The character data acquisition unit 20, the video splitting unit 21, the face matching result acquisition unit 22 and the character search result acquisition unit 23 are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to split the video data into single-frame images, and use the image character search model to accurately match the face of each frame of the image, and then the matching results of all frames are combined to generate the final character search results, so as to quickly and accurately search for video frames with non-compliant character portraits in the video, thereby improving the processing efficiency of the media platform for related video frames and ensuring the compliance of the media content.

[0154] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training method of the image person search model and / or the image person search method.

[0155] An embodiment of the present invention provides a processor, which is used to run a program, wherein when the program is running, the training method of the image person search model and / or the image person search method are executed.

[0156] like Fig.10 As shown, an embodiment of the present invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the above-mentioned image person search model training method and / or image person search method. The electronic device in this article can be a server, PC, PAD, mobile phone, etc.

[0157] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a training method for initializing an image person search model and / or a program of steps of an image person search method.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0160] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0161] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0163] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0164] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present invention.

[0165] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0167] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A training method for an image person search model, characterized in that: include: Obtaining a pre-constructed face image dataset, wherein the face image dataset includes at least one group of face image data, each group of face image data includes a base face image, a target image, and a sample label, and the sample label is used to indicate whether the target image contains face information in the base face image; For any group of face image data in the face image data set: input the base face image and the target image of the face image data into an image person search model, extract face feature maps of the base face image at multiple different scales, and extract output feature maps of the target image at multiple different scales; Using the face feature map and the output feature map to perform feature comparison, obtaining similarity feature maps between the base face image and the target image at multiple different scales; Performing feature integration on the similarity feature maps at multiple different scales, and outputting a face matching result of the face image data; The face matching results of each group of face image data are used to adjust the model parameters of the image person search model based on a preset loss function to obtain the trained image person search model.

2. The method according to claim 1, characterized in that The method of performing feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales includes: The facial feature maps at multiple different scales are respectively used as convolution kernels, and then convolution operations are performed with the output feature maps at multiple different scales to obtain similarity feature maps between the base face image and the target image at multiple different scales.

3. The method according to claim 1, characterized in that The step of integrating the similarity feature graphs at multiple different scales and outputting the face matching result of the face image data includes: Downsampling and channel-connecting the similarity feature maps at multiple different scales to obtain a feature information integration feature map; The feature information is integrated with the feature graph for full connection, and the face matching result of the face image data is output.

4. The method according to claim 1, characterized in that: The step of extracting facial feature maps of the base face image at multiple scales includes: Extracting a first facial feature map of the base facial image at a first preset scale using a first pre-trained feature extraction network; Downsampling the first facial feature map to obtain a second facial feature map of the base facial image at a second preset scale; The second facial feature map is downsampled to obtain a third facial feature map of the base facial image at a third preset scale.

5. The method according to claim 4, characterized in that The first preset scale is n×c×7×7, where n is the number of training samples used in each parameter update, and c is the number of channels of the feature map.

6. The method according to claim 1, characterized in that The step of extracting output feature maps of the target image at multiple scales includes: Extracting a first effective feature map of the target image at a fourth preset scale using a second pre-trained feature extraction network; Downsampling and rotating the first effective feature map to capture features, to obtain a second effective feature map of the target image at a fifth preset scale; Downsampling and rotating the second effective feature map to capture features, to obtain a third effective feature map of the target image at a sixth preset scale; Using the third effective feature map as a third output feature map, upsampling the third output feature map and performing feature fusion with the second effective feature map to obtain a second output feature map; Upsampling the third output feature map and performing feature fusion with the second effective feature map to obtain a second output feature map; The second output feature map is upsampled and then feature fused with the first effective feature map to obtain a first output feature map.

7. A method for searching for people in an image, characterized in that: include: Obtain video data and a base face image of a person to be searched; Splitting the video data into single-frame video images; For any of the single-frame video images: inputting the single-frame video image and the base face image into the image person search model of any one of claims 1 to 6 to obtain a face matching result output by the image person search model; Based on the face matching results of each of the single-frame video images, a person search result of the video data is generated.

8. A training device for an image person search model, characterized in that: include: face image data set acquisition unit, feature extraction unit, feature comparison unit, feature integration unit and model parameter adjustment unit, The face image data set obtaining unit is used to obtain a pre-constructed face image data set, wherein the face image data set includes at least one group of face image data, each group of face image data includes a base face image, a target image and a sample label, and the sample label is used to indicate whether the target image contains face information in the base face image; The feature extraction unit is used for: for any group of the face image data in the face image data set: inputting the base face image and the target image of the face image data into the image person search model, extracting face feature maps of the base face image at multiple different scales, and extracting output feature maps of the target image at multiple different scales; The feature comparison unit is used to perform feature comparison using the face feature map and the output feature map to obtain similarity feature maps between the base face image and the target image at multiple different scales; The feature integration unit is used to perform feature integration on the similarity feature maps at multiple different scales and output a face matching result of the face image data; The model parameter adjustment unit is used to use the face matching results of each group of face image data to adjust the model parameters of the image person search model based on a preset loss function to obtain the trained image person search model.

9. An image person search device, characterized in that: include: A character data acquisition unit, a video splitting unit, a face matching result acquisition unit and a character search result acquisition unit, The character data obtaining unit is used to obtain the video data and the base face image of the character to be searched; The video splitting unit is used to split the video data into single-frame video images; The face matching result obtaining unit is used for: for any of the single-frame video images: inputting the single-frame video image and the base face image into the image character search model according to claim 8, and obtaining the face matching result output by the image character search model; The person search result obtaining unit is used to generate a person search result of the video data based on the face matching result of each of the single-frame video images.

10. An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein: The processor and the memory communicate with each other via the bus; The processor is used to call the program instructions in the memory to execute the training method of the image person search model described in any one of claims 1 to 6, and / or to execute the image person search method described in claim 7.

Images