Image screening method and device, image recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides an image screening method and device, an image recognition method and device and electronic equipment, and relates to the technical field of computer vision. The method comprises the following steps: selecting an image which is not screened currently from all images to be screened as a current image to be screened; inputting the current to-be-screened image into a pre-trained quality evaluation model to obtain predicted sub-scores, and calculating a predicted total score of the current to-be-screened image according to each predicted sub-score and the current weight of each quality evaluation dimension represented by each branch network; and if the predicted total score of the current to-be-screened image belongs to a preset range, adjusting the current weight of each quality evaluation dimension based on a reinforcement learning algorithm according to the current filtering proportion and the target filtering proportion, and if the predicted total score of the current to-be-screened image is not less than a preset filtering threshold value, carrying out quality evaluation on the current to-be-screened image. And determining that the screening result of the current to-be-screened image is reserved. Therefore, the image screening efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image screening method, an image recognition method, a device and an electronic device. Background Art

[0002] With the development of computer vision technology, the technology of detecting objects in images by matching image features has been widely used. Accordingly, a part of labeled images are usually pre-stored in electronic devices. For example, electronic devices can store labeled images through a preset image library. Among them, the label of any image represents the identity of the object in the image. Furthermore, in practical applications, when an electronic device receives an image to be identified, it can use the image features of the image to be identified to compare with the image features of the pre-stored image to determine the identity of the object in the image to be identified. If the quality of the pre-stored image is not high, the recognition result of the image to be identified may be inaccurate. In one method, the images can be screened manually to select high-quality images for storage.

[0003] However, the efficiency of screening by manual method is not high. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image screening method, an image recognition method, a device and an electronic device to improve the efficiency of image screening. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a method for image screening is first provided, the method comprising:

[0006] Select an image that is not currently being screened from all the images to be screened as the current image to be screened;

[0007] The current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and the predicted total score of the current image to be screened is calculated according to each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein the quality assessment model is obtained by training based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality assessment dimension;

[0008] If the predicted total score of the current image to be screened belongs to the preset range, the current weights of each quality evaluation dimension are adjusted based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio, and when the predicted total score of the current image to be screened is not less than the preset filtering threshold, the screening result of the current image to be screened is determined to be retained, and when the predicted total score of the current image to be screened is less than the preset filtering threshold, the screening result of the current image to be screened is determined to be not retained; wherein, the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose screening results are not retained to the number of images whose screening results have been determined at present;

[0009] Returning to the step of selecting an image that is not currently being screened from all the images to be screened as the current image to be screened, until the screening results of all the images to be screened are obtained;

[0010] Based on the images that are retained as a result of the screening, images for adding to the current image library are obtained.

[0011] In some embodiments, obtaining images for adding to the current image library based on the images that are retained based on the screening results includes:

[0012] Select an image that is not currently processed from the images that are retained in the screening results as the current image to be processed;

[0013] Extract features of the current image to be processed to obtain the current first image features;

[0014] Based on the matching results of the current first feature clusters and the current first image features, determining whether the current image to be processed contains the specified object; wherein the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; and the images in the current first image set do not contain the specified object;

[0015] If the current image to be processed contains the specified object, the current image to be processed is determined to be an image to be added to the current image library, and the step of returning to execute the step of selecting a currently unprocessed image from the images retained as a result of the screening as the current image to be processed.

[0016] In some embodiments, determining whether the current image to be processed contains the specified object based on the matching results of the current first feature clusters and the current first image features includes:

[0017] Calculating first similarities between the current first image feature and the cluster centers of the current first feature clusters;

[0018] From the current first feature clusters, select the first feature cluster with the largest first similarity;

[0019] Calculating each second similarity between the current first image feature and each image feature included in the currently selected first feature cluster;

[0020] According to each second similarity corresponding to the current image to be processed, it is determined whether the current image to be processed contains the designated object.

[0021] In some embodiments, determining whether the current image to be processed contains the specified object according to each second similarity corresponding to the current image to be processed includes:

[0022] Determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed;

[0023] If the first preset number of largest second similarities are all greater than the first preset similarity threshold, it is determined that the current image to be processed does not contain the designated object; otherwise, it is determined that the current image to be processed contains the designated object.

[0024] In some embodiments, the method further comprises:

[0025] When it is determined that the current image to be processed does not contain the designated object, the current image to be processed is added to the current first image set.

[0026] In some embodiments, before inputting the current image to be screened into a pre-trained quality assessment model to obtain the predicted sub-score output by each branch network in the quality assessment model, the method further includes:

[0027] Perform feature extraction on the current image to be screened to obtain the current second image feature;

[0028] Based on the matching results of the current second feature clusters and the current second image features, determining whether the current image to be screened contains the specified object; wherein the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; and the images in the current second image set do not contain the specified object;

[0029] The step of inputting the current image to be screened into a pre-trained quality assessment model to obtain the prediction sub-score output by each branch network in the quality assessment model comprises:

[0030] In the case that the current image to be screened contains the designated object, the current image to be screened is input into a pre-trained quality assessment model to obtain a prediction sub-score output by each branch network in the quality assessment model.

[0031] In some embodiments, determining whether the current image to be screened contains the specified object based on the matching results of the current second feature clusters and the current second image features includes:

[0032] Calculating third similarities between the current second image feature and the cluster centers of the current second feature clusters;

[0033] From the current second feature clusters, select the second feature cluster with the largest third similarity;

[0034] Calculating each fourth similarity between the current second image feature and each image feature included in the currently selected second feature cluster;

[0035] It is determined whether the current image to be screened contains the designated object according to each fourth similarity corresponding to the current image to be screened.

[0036] In some embodiments, determining whether the current image to be screened contains the specified object according to each fourth similarity corresponding to the current image to be screened includes:

[0037] Determine a second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened;

[0038] If the second preset number of largest fourth similarities are all greater than the second preset similarity threshold, it is determined that the current image to be screened does not contain the specified object; otherwise, it is determined that the current image to be screened contains the specified object.

[0039] In some embodiments, the method further comprises:

[0040] When it is determined that the current image to be filtered does not contain the specified object, the current image to be filtered is added to the current second image set, and the step of selecting an image that is not currently filtered from all the images to be filtered is returned to be executed as the current image to be filtered.

[0041] In some embodiments, adjusting the current weights of the quality evaluation dimensions based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio includes:

[0042] Get the current status; the current status includes: the current filtering ratio, the number of images for which the filtering results have been determined, and the current weight of each quality evaluation dimension;

[0043] Determine the expected benefit when taking each alternative action in the current state as the current first expectation; wherein an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension;

[0044] Select the alternative action corresponding to the current largest first expectation as the action to be taken currently;

[0045] Based on the current actions that need to be taken, the current weights of each quality evaluation dimension are adjusted.

[0046] In some embodiments, the method further comprises:

[0047] After determining the action currently required to be taken, updating the current first expectation based on the first formula;

[0048] Among them, the first formula is:

[0049] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0050]

[0051] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken at present; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after the screening result of the current image to be screened is obtained, maxQ(S′,A′) represents the maximum value of the second expectation of the benefit obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value in the second expectation;

[0052] st represents the constraint condition, R(S,A) represents: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c represents the preset filtering threshold, R min Indicates the minimum value in the preset range, R max represents the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other situations.

[0053] In some embodiments, the quality assessment model is trained by the following steps:

[0054] Acquire a sample image containing a specified object; wherein the label of the sample image comprises: a sample sub-score of the sample image in each quality evaluation dimension, and a sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and weights of each quality evaluation dimension of the sample image;

[0055] For each sample image, the sample image is input into the quality evaluation model of the initial structure to obtain the training sub-scores output by each branch network in the quality evaluation model of the initial structure, and the total training score of the sample image is calculated according to the initial values ​​of the weights of each training sub-score and each quality evaluation dimension represented by each branch network;

[0056] Based on each training sub-score and the total training score of the sample image, as well as each sample sub-score and the total sample score of the sample image, a loss value is calculated;

[0057] Based on the obtained loss value, the model parameters of the quality evaluation model of the initial structure are adjusted until convergence is achieved.

[0058] In some embodiments, the calculation of the predicted total score of the current image to be screened according to each predicted sub-score and the current weight of each quality evaluation dimension represented by each branch network includes:

[0059] According to each prediction sub-score and the current weight of each quality evaluation dimension represented by each branch network, the predicted total score of the current image to be screened is calculated according to the second formula;

[0060] Wherein, the second formula is:

[0061] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0062] f(z) represents the predicted total score of the current image to be screened; k represents the number of quality evaluation dimensions; f(x1) represents the predicted sub-score of the current image to be screened on the first quality evaluation dimension, and α represents the current weight of the first quality evaluation dimension; f(x i ) represents the predicted sub-score of the current image to be screened on the i-th quality evaluation dimension, β represents the current weight of the i-th quality evaluation dimension; f(x k ) represents the predicted sub-score of the current image to be screened on the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0063] In some embodiments, the quality evaluation dimensions include at least one of the following: image clarity, image visibility, a pitch angle of a specified object in the image, a heading angle of a specified object in the image, and an area ratio of a specified object in the image.

[0064] A second aspect of the embodiments of the present application provides an image recognition method, the method comprising:

[0065] Obtain an image to be recognized;

[0066] Extracting features of the image to be identified to obtain features of the image to be identified;

[0067] Based on the matching result between the features of the image to be identified and the image features of the images in the current image library, the identification result of the image to be identified is obtained; wherein, the images in the current image library are obtained based on any image screening method described in the first aspect above.

[0068] In some embodiments, before obtaining the recognition result of the image to be recognized based on the matching result between the image features to be recognized and the image features of the images in the current image library, the method further includes:

[0069] Calculating the fifth similarities between the image feature to be identified and the cluster center of each current third feature cluster; wherein each current third feature cluster is obtained by clustering the image features of the images included in the current third image set; and the images in the current third image set do not include the specified object;

[0070] From the current third feature clusters, select the corresponding third feature cluster with the largest fifth similarity;

[0071] Calculating each sixth similarity between the image feature to be identified and each image feature included in the currently selected third feature cluster;

[0072] determining whether the image to be identified contains a specified object according to each sixth similarity corresponding to the image to be identified;

[0073] The obtaining the recognition result of the image to be recognized based on the matching result between the image features to be recognized and the image features of the images in the current image library includes:

[0074] In the case that the image to be recognized contains a designated object, a recognition result of the image to be recognized is obtained based on a matching result between features of the image to be recognized and image features of images in the current image library.

[0075] In some embodiments, determining whether the image to be identified contains a specified object according to each sixth similarity corresponding to the image to be identified includes:

[0076] Determine a third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified;

[0077] If the third preset number of largest sixth similarities are all greater than the third preset similarity threshold, it is determined that the image to be identified does not contain the designated object; otherwise, it is determined that the image to be identified contains the designated object.

[0078] In some embodiments, the method further comprises:

[0079] In the case that the image to be recognized does not contain the designated object, the image to be recognized is added to the current third image set.

[0080] According to a third aspect of the embodiments of the present application, an image screening device is provided, the device comprising:

[0081] A first image selection module is used to select an image that is not currently being screened from all images to be screened as the current image to be screened;

[0082] The first scoring module is used to input the current image to be screened into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the current image to be screened according to each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein the quality assessment model is obtained by training based on the sample sub-scores of the sample images containing the specified objects on each quality assessment dimension and the sample total score of the sample images; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality assessment dimension;

[0083] A screening result determination module is used to adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio if the predicted total score of the current image to be screened belongs to a preset range, and to determine that the screening result of the current image to be screened is retained when the predicted total score of the current image to be screened is not less than the preset filtering threshold, and to determine that the screening result of the current image to be screened is not retained when the predicted total score of the current image to be screened is less than the preset filtering threshold; wherein the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose screening results are not retained to the number of images whose screening results have been determined at present; and return to trigger the first image selection module until the screening results of all images to be screened are obtained;

[0084] The image adding module is used to obtain images to be added to the current image library based on the images that are retained as a result of the screening.

[0085] In some embodiments, the image adding module includes:

[0086] A first image selection submodule is used to select a currently unprocessed image from the images that are retained in the screening results as a current image to be processed;

[0087] A first feature extraction submodule is used to extract features of the current image to be processed to obtain a current first image feature;

[0088] A first object determination submodule is used to determine whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features; wherein the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; and the images in the current first image set do not contain the specified object;

[0089] The first image adding submodule is used to determine that the current image to be processed is an image to be added to the current image library if the current image to be processed contains the designated object, and trigger the first image selecting submodule.

[0090] In some embodiments, the first object determination submodule includes:

[0091] A first similarity calculation unit, used for calculating first similarities between a current first image feature and a cluster center of each current first feature cluster;

[0092] A first feature cluster selection unit, configured to select a first feature cluster with the largest first similarity from among the current first feature clusters;

[0093] A second similarity calculation unit, used for calculating each second similarity between the current first image feature and each image feature included in the currently selected first feature cluster;

[0094] The first object determination unit is used to determine whether the current image to be processed contains a designated object according to each second similarity corresponding to the current image to be processed.

[0095] In some embodiments, the first object determination unit is specifically configured to:

[0096] Determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed;

[0097] If the first preset number of largest second similarities are all greater than the first preset similarity threshold, it is determined that the current image to be processed does not contain the designated object; otherwise, it is determined that the current image to be processed contains the designated object.

[0098] In some embodiments, the apparatus further comprises:

[0099] The first image set updating module is used to add the current image to be processed to the current first image set when it is determined that the current image to be processed does not contain the specified object.

[0100] In some embodiments, the apparatus further comprises:

[0101] A second feature extraction module is used to extract features from the current image to be screened to obtain a current second image feature before inputting the current image to be screened into a pre-trained quality assessment model to obtain a predicted sub-score output by each branch network in the quality assessment model;

[0102] A first object determination module is used to determine whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features; wherein the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; and the images in the current second image set do not contain the specified object;

[0103] The first scoring module is specifically used for:

[0104] In the case that the current image to be screened contains the designated object, the current image to be screened is input into a pre-trained quality assessment model to obtain a prediction sub-score output by each branch network in the quality assessment model.

[0105] In some embodiments, the first object determination module includes:

[0106] A first similarity calculation submodule, used for calculating third similarities between the current second image feature and the cluster center of each current second feature cluster;

[0107] A first feature cluster selection submodule is used to select a second feature cluster with the largest third similarity from the current second feature clusters;

[0108] A second similarity calculation submodule, used for calculating each fourth similarity between the current second image feature and each image feature included in the currently selected second feature cluster;

[0109] The second object determination submodule is used to determine whether the current image to be screened contains a specified object according to each fourth similarity corresponding to the current image to be screened.

[0110] In some embodiments, the second object determination submodule is specifically configured to:

[0111] Determine a second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened;

[0112] If the second preset number of largest fourth similarities are all greater than the second preset similarity threshold, it is determined that the current image to be screened does not contain the specified object; otherwise, it is determined that the current image to be screened contains the specified object.

[0113] In some embodiments, the apparatus further comprises:

[0114] The second image set updating module is used to add the current image to be screened to the current second image set and trigger the first image selection module when it is determined that the current image to be screened does not contain the specified object.

[0115] In some embodiments, the screening result determination module is specifically used to:

[0116] Get the current status; the current status includes: the current filtering ratio, the number of images for which the filtering results have been determined, and the current weight of each quality evaluation dimension;

[0117] Determine the expected benefit when taking each alternative action in the current state as the current first expectation; wherein an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension;

[0118] Select the alternative action corresponding to the current largest first expectation as the action to be taken currently;

[0119] Based on the current actions that need to be taken, the current weights of each quality evaluation dimension are adjusted.

[0120] In some embodiments, the apparatus further comprises:

[0121] An expectation updating module, configured to update a current first expectation based on a first formula after determining the action currently required to be taken;

[0122] Among them, the first formula is:

[0123] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0124]

[0125] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken at present; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after the screening result of the current image to be screened is obtained, maxQ(S′,A′) represents the maximum value of the second expectation of the benefit obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value in the second expectation;

[0126] st represents the constraint condition, R(S,A) represents: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c represents the preset filtering threshold, R min Indicates the minimum value in the preset range, R max represents the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other situations.

[0127] In some embodiments, the quality assessment model is trained by the following steps:

[0128] Acquire a sample image containing a specified object; wherein the label of the sample image comprises: a sample sub-score of the sample image in each quality evaluation dimension, and a sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and weights of each quality evaluation dimension of the sample image;

[0129] For each sample image, the sample image is input into the quality evaluation model of the initial structure to obtain the training sub-scores output by each branch network in the quality evaluation model of the initial structure, and the total training score of the sample image is calculated according to the initial values ​​of the weights of each training sub-score and each quality evaluation dimension represented by each branch network;

[0130] Based on each training sub-score and the total training score of the sample image, as well as each sample sub-score and the total sample score of the sample image, a loss value is calculated;

[0131] Based on the obtained loss value, the model parameters of the quality evaluation model of the initial structure are adjusted until convergence is achieved.

[0132] In some embodiments, the first scoring module is specifically used to:

[0133] According to each prediction sub-score and the current weight of each quality evaluation dimension represented by each branch network, the predicted total score of the current image to be screened is calculated according to the second formula;

[0134] Wherein, the second formula is:

[0135] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0136] f(z) represents the predicted total score of the current image to be screened; k represents the number of quality evaluation dimensions; f(x1) represents the predicted sub-score of the current image to be screened on the first quality evaluation dimension, and α represents the current weight of the first quality evaluation dimension; f(x i ) represents the predicted sub-score of the current image to be screened on the i-th quality evaluation dimension, β represents the current weight of the i-th quality evaluation dimension; f(x k ) represents the predicted sub-score of the current image to be screened on the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0137] In some embodiments, the quality evaluation dimensions include at least one of the following: image clarity, image visibility, a pitch angle of a specified object in the image, a heading angle of a specified object in the image, and an area ratio of a specified object in the image.

[0138] According to a fourth aspect of the embodiments of the present application, an image recognition device is provided, the device comprising:

[0139] An image acquisition module to be identified, used to acquire an image to be identified;

[0140] A first feature extraction module is used to extract features of the image to be identified to obtain features of the image to be identified;

[0141] A recognition result determination module is used to obtain the recognition result of the image to be recognized based on the matching result between the image features to be recognized and the image features of the images in the current image library; wherein the images in the current image library are obtained based on the image screening method described in the first aspect above.

[0142] In some embodiments, the apparatus further comprises:

[0143] A first similarity calculation module is used to calculate the fifth similarities between the image feature to be identified and the cluster centers of the current third feature clusters before obtaining the recognition result of the image to be identified based on the matching result between the image feature to be identified and the image features of the images in the current image library; wherein the current third feature clusters are obtained by clustering the image features of the images included in the current third image set; and the images in the current third image set do not include the specified object;

[0144] A first feature cluster selection module is used to select a third feature cluster with the largest fifth similarity from the current third feature clusters;

[0145] A second similarity calculation module, used for calculating each sixth similarity between the image feature to be identified and each image feature included in the currently selected third feature cluster;

[0146] A second object determination module, configured to determine whether the image to be identified contains a specified object according to each sixth similarity corresponding to the image to be identified;

[0147] The recognition result determination module is specifically used for:

[0148] In the case that the image to be recognized contains a designated object, a recognition result of the image to be recognized is obtained based on a matching result between features of the image to be recognized and image features of images in the current image library.

[0149] In some embodiments, the second object determination module is specifically configured to:

[0150] Determine a third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified;

[0151] If the third preset number of largest sixth similarities are all greater than the third preset similarity threshold, it is determined that the image to be identified does not contain the designated object; otherwise, it is determined that the image to be identified contains the designated object.

[0152] In some embodiments, the apparatus further comprises:

[0153] The third image set updating module is used to add the image to be identified to the current third image set when the image to be identified does not contain the specified object.

[0154] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including:

[0155] Memory, used to store computer programs;

[0156] The processor is used to implement any of the above-mentioned image screening methods or image recognition methods when executing the program stored in the memory.

[0157] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned image screening methods or image recognition methods.

[0158] In another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image screening methods or image recognition methods.

[0159] Beneficial effects of the embodiments of the present application:

[0160] Based on the image screening method provided by the embodiment of the present application, an image that is not currently screened can be selected from all images to be screened as the current image to be screened, and the current image to be screened can be input into a pre-trained quality assessment model. The quality assessment model includes multiple branch networks, and a branch network represents a quality assessment dimension. Accordingly, the quality assessment model can output the sub-scores (i.e., predicted sub-scores) of the current image to be screened on each quality assessment dimension. Furthermore, the predicted total score of the current image to be screened can be calculated based on the current weights of each predicted sub-score and each quality assessment dimension. According to the predicted total score of the current image to be screened, the screening result (retained or not) of the current image to be screened is determined. Accordingly, the screening result of each image in all images to be screened can be obtained according to the above steps. In this way, automatic scoring and screening for each image to be screened can be achieved through a pre-trained quality assessment model, without the need to manually screen the image to be screened, thereby improving the screening efficiency.

[0161] After obtaining the screening results of each image to be screened, since the screening results are determined based on the predicted total score of the image, the predicted total score of the image can reflect the overall quality of the image in multiple quality evaluation dimensions. Therefore, based on the images that are retained in the screening results, the images to be added to the current image library are obtained, which can ensure that images with higher quality are added to the current image library for subsequent processing.

[0162] In addition, in the process of screening images, the target filtering ratio and the current filtering ratio can be further combined to dynamically adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm. In other words, in each stage of screening all images to be screened, the current filtering ratio can be continuously adjusted to the target filtering ratio. In this way, all images to be screened can be screened according to actual filtering needs, and the ratio between the number of images retained as a result of the screening and the number of images not retained as a result of the screening can be achieved, which meets the personalized needs of users.

[0163] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0164] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0165] Figure 1 A first flow chart of the image screening method provided in the embodiment of the present application;

[0166] Figure 2 A second flow chart of the image screening method provided in an embodiment of the present application;

[0167] Figure 3 A flowchart for determining whether a current image to be processed contains a specified object is provided in an embodiment of the present application;

[0168] Figure 4 A third flow chart of the image screening method provided in an embodiment of the present application;

[0169] Figure 5 A flowchart for adjusting the current weight of each quality evaluation dimension provided in an embodiment of the present application;

[0170] Figure 6 A flowchart of a training quality evaluation model provided in an embodiment of the present application;

[0171] Figure 7 A first flow chart of the image recognition method provided in the embodiment of the present application;

[0172] Figure 8 A second flow chart of the image recognition method provided in the embodiment of the present application;

[0173] Fig. 9 A schematic diagram of a process for detecting an image provided in an embodiment of the present application;

[0174] Fig.10 A structural diagram of an image screening device provided in an embodiment of the present application;

[0175] Fig.11 A structural diagram of an image recognition device provided in an embodiment of the present application;

[0176] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0177] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0178] With the development of computer vision technology, the technology of detecting objects in images by matching image features has been widely used. Accordingly, a part of labeled images are usually pre-stored in electronic devices. For example, electronic devices can store labeled images through a preset image library. Among them, the label of any image represents the identity of the object in the image. Furthermore, in practical applications, when an electronic device receives an image to be identified, it can use the image features of the image to be identified to compare with the image features of the pre-stored image to determine the identity of the object in the image to be identified. If the quality of the pre-stored image is not high, the recognition result of the image to be identified may be inaccurate. In one method, the images can be screened manually to select high-quality images for storage.

[0179] However, the efficiency of screening by manual method is not high.

[0180] The image screening method provided in the present application can be applied to application scenarios such as identity authentication and type recognition. Accordingly, the types of specified objects in different application scenarios may be different.

[0181] For example, in an identity authentication scenario, the designated object may be a person's face. Accordingly, a user (such as a community owner) may upload a facial photo of a person who has the authority to enter a designated place (such as a community) and the person's identity information in advance through a smart terminal, and accordingly, the electronic device may save the photo to a preset image library. Subsequently, the electronic device may obtain an image of the face of the person who currently needs to enter the designated place as an image to be identified through a gate machine set at the entrance of the designated place. Furthermore, the electronic device may compare the image features of the acquired image to be identified with the image features of each image in the image library to determine whether the person who currently needs to enter the designated place has the authority to enter the designated place.

[0182] For another example, in an identity authentication scenario, the designated object may be a human body. Accordingly, a user (such as a community owner) may upload a full-body photo of a person who has the authority to enter a designated place (such as a community) and the identity information of the person in advance through a smart terminal, and accordingly, the electronic device may save the photo to a preset image library. Subsequently, the electronic device may obtain an image of the person who currently needs to enter the designated place as an image to be identified through a gate machine set at the entrance of the designated place. Furthermore, the electronic device may compare the image features of the acquired image to be identified with the image features of each image in the image library to determine whether the person who currently needs to enter the designated place has the authority to enter the designated place.

[0183] It is understandable that the images uploaded by users may have problems such as blur, distortion, too small an object ratio, etc., that is, the image quality is not high. In turn, the recognition result of the image to be recognized may be inaccurate.

[0184] The present application embodiment provides an image screening method, which can be applied to electronic devices. Figure 1 , Figure 1 The first flow chart of the image screening method provided in the embodiment of the present application comprises the following steps:

[0185] S101: Selecting an image that is not currently being screened from all images to be screened as the current image to be screened.

[0186] S102: Input the current image to be screened into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the current image to be screened based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network.

[0187] Among them, the quality evaluation model is obtained by training based on the sample sub-scores of the sample images containing the specified objects in each quality evaluation dimension and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality evaluation dimension.

[0188] S103: If the predicted total score of the current image to be screened belongs to the preset range, the current weights of each quality evaluation dimension are adjusted based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio, and when the predicted total score of the current image to be screened is not less than the preset filtering threshold, the screening result of the current image to be screened is determined to be retained, and when the predicted total score of the current image to be screened is less than the preset filtering threshold, the screening result of the current image to be screened is determined to be not retained.

[0189] If the screening results of all the images to be screened have not been obtained currently, the process returns to step S101 ; if the screening results of all the images to be screened have been obtained currently, the process proceeds to step S104 .

[0190] The preset range includes a preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose filtering results are not retained to the number of images whose filtering results have been determined currently.

[0191] S104: Based on the images that are retained as a result of the screening, images for adding to the current image library are obtained.

[0192] Based on the above processing, an image that is not currently screened can be selected from all the images to be screened as the current image to be screened, and the current image to be screened can be input into the pre-trained quality evaluation model. The quality evaluation model includes multiple branch networks, and one branch network represents a quality evaluation dimension. Accordingly, the quality evaluation model can output the sub-scores (i.e., predicted sub-scores) of the current image to be screened on each quality evaluation dimension. Furthermore, the predicted total score of the current image to be screened can be calculated based on the current weights of each predicted sub-score and each quality evaluation dimension. According to the predicted total score of the current image to be screened, the screening result (retain or not) of the current image to be screened is determined. Accordingly, the screening result of each image in all the images to be screened can be obtained according to the above steps. In this way, automatic scoring and screening for each image to be screened can be achieved through the pre-trained quality evaluation model, without the need to manually screen the images to be screened, thereby improving the screening efficiency.

[0193] After obtaining the screening results of each image to be screened, since the screening results are determined based on the predicted total score of the image, the predicted total score of the image can reflect the overall quality of the image in multiple quality evaluation dimensions. Therefore, based on the images that are retained in the screening results, the images to be added to the current image library are obtained, which can ensure that images with higher quality are added to the current image library for subsequent processing.

[0194] In addition, in the process of screening images, the target filtering ratio and the current filtering ratio can be further combined to dynamically adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm. In other words, in each stage of screening all images to be screened, the current filtering ratio can be continuously adjusted to the target filtering ratio. In this way, all images to be screened can be screened according to actual filtering needs, and the ratio between the number of images retained as a result of the screening and the number of images not retained as a result of the screening can be achieved, which meets the personalized needs of users.

[0195] With respect to step S101 , the image to be screened represents: an image acquired by the electronic device and to be added to the image library.

[0196] For ease of description, the above-mentioned identity authentication scenario is taken as an example. In the above-mentioned identity authentication scenario, the designated object is a person's face, and the image to be screened can be: photos of people who have the authority to enter the community uploaded by the community owner through the smart terminal.

[0197] In one implementation, during the process of performing image screening according to the above steps S101 to S104, the electronic device can acquire new images to be screened in real time. That is, the total number of images to be screened is variable during the image screening process. In this way, during the image screening process, the electronic device can screen the images acquired in real time, thereby improving the flexibility of image screening. The specific process of performing image screening will be described in subsequent embodiments.

[0198] In another implementation, the electronic device may also acquire all images to be added to the image library at one time to obtain all images to be screened. That is, the total number of images to be screened is fixed during the image screening process.

[0199] Furthermore, the electronic device may select an image that is not currently being screened from all the images to be screened as the current image to be screened.

[0200] In the embodiment of the present application, the electronic device may determine the current image to be screened according to the order in which the images to be screened are acquired, and process it according to steps S102 to S103. Alternatively, the electronic device may randomly select one of all the images to be screened that are not currently screened as the current image to be screened, and process it according to steps S102 to S103. The specific method of determining the current image to be screened is not limited herein.

[0201] With respect to step S102, after determining the current image to be screened, the electronic device can input the current image to be screened into a pre-trained quality assessment model. The quality assessment model includes multiple branch networks, and each branch network represents a quality assessment dimension. The specific training process of the quality assessment model will be described in subsequent embodiments.

[0202] The quality assessment model can process the input image to be screened, and then each branch network in the quality assessment model can output the predicted sub-score of the image to be screened on the quality assessment dimension represented by each.

[0203] In one implementation, each quality evaluation dimension includes at least one of the following: image clarity, image visibility, a pitch angle of a specified object in the image, a heading angle of a specified object in the image, and an area ratio of a specified object in the image.

[0204] Among them, the clarity of the image indicates whether the details, textures and colors in the image are clear and sharp, that is, the degree of recognizability of the image content.

[0205] Correspondingly, if the clarity of an image is high, the sub-score of the image on the quality evaluation dimension (image clarity) is high; if the clarity of an image is low, the sub-score of the image on the quality evaluation dimension (image clarity) is low.

[0206] The visibility of an image indicates the degree of occlusion of a specified object in the image. For example, when the specified object is a person's face, the degree of occlusion of the specified object in the image is the proportion of the person's face that is occluded in the image. In other words, the greater the proportion of the person's face that is occluded in the image, the higher the degree of occlusion of the specified object in the image, and the lower the visibility of the image.

[0207] Correspondingly, if the visibility of an image is high, the sub-score of the image on the quality evaluation dimension (visibility of the image) is high; if the visibility of an image is low, the sub-score of the image on the quality evaluation dimension (visibility of the image) is low.

[0208] The pitch angle of the specified object in the image represents: in the height direction, the distance of the lens of the image acquisition device (such as a camera) that acquires the image relative to the specified object. It can be understood that when the lens of the image acquisition device and the specified object are at the same height, the pitch angle of the specified object in the image is 0; the greater the distance of the lens of the image acquisition device relative to the specified object in the height direction, the greater the pitch angle of the specified object in the image. The smaller the pitch angle of the specified object in the image, the smaller the distortion effect of the lens distortion on the specified object in the image, and the larger the pitch angle of the specified object in the image, the greater the distortion effect of the lens distortion on the specified object in the image.

[0209] Correspondingly, if the pitch angle of a specified object in an image is small, the sub-score of the image on the quality evaluation dimension (the pitch angle of the specified object in the image) is high; if the pitch angle of a specified object in an image is large, the sub-score of the image on the quality evaluation dimension (the pitch angle of the specified object in the image) is low.

[0210] The heading angle of the specified object in the image represents: in the horizontal direction, the angle between the optical axis of the lens of the image acquisition device (such as a camera) that acquires the image and the specified plane of the specified object. For example, when the specified object is a person's face, the specified plane may be: the plane where the person's face is approximately located. That is to say, in the horizontal direction, the closer the position of the image acquisition device is to the front of the specified object, the smaller the heading angle of the specified object in the image. The smaller the heading angle of the specified object in the image, the more complete the facial information of the person's face is collected, and the larger the heading angle of the specified object in the image, the more incomplete the facial information of the person's face is collected.

[0211] Correspondingly, if the heading angle of a specified object in an image is small, the sub-score of the image on the quality evaluation dimension (the heading angle of the specified object in the image) is high; if the heading angle of a specified object in an image is large, the sub-score of the image on the quality evaluation dimension (the heading angle of the specified object in the image) is low.

[0212] The area ratio of the specified object in the image indicates: the ratio of the image area occupied by the specified object in the image to the image area of ​​the complete image. It is understandable that other image areas other than the image area occupied by the specified object in the image usually represent invalid information (such as background information), etc. Therefore, the larger the area ratio of the specified object in the image, the larger the ratio of valid information in the image; the smaller the area ratio of the specified object in the image, the smaller the ratio of valid information in the image.

[0213] Correspondingly, if the area proportion of a specified object in an image is large, the sub-score of the image on this quality evaluation dimension (the area proportion of the specified object in the image) is high; if the area proportion of a specified object in an image is small, the sub-score of the image on this quality evaluation dimension (the area proportion of the specified object in the image) is low.

[0214] In this way, the quality assessment model can evaluate the image quality of the image input to the model from multiple quality assessment dimensions, that is, output the sub-score of the image on the quality assessment dimension. In this way, it avoids evaluating the image quality from a single dimension and improves the accuracy of the image quality assessment. In turn, the image quality of the stored images can be guaranteed.

[0215] After obtaining the predicted sub-scores of the current image to be screened in each quality evaluation dimension, the electronic device can calculate the predicted total score of the current image to be screened according to each predicted sub-score and the current weight of each quality evaluation dimension represented by each branch network. Among them, the current weight of each quality evaluation dimension is: adjusted on the initial value of the weight of each quality evaluation dimension, and the specific process of adjusting the weight of each quality evaluation dimension will be described in the subsequent embodiments.

[0216] The current weight range of each quality evaluation dimension is [0,1], that is, the predicted total score of the current image to be screened is positively correlated with the predicted sub-score of the image in each quality evaluation dimension.

[0217] In some embodiments, the electronic device may calculate a weighted sum of the predicted sub-scores according to the current weights of the predicted sub-scores and the quality assessment dimensions as the predicted total score of the current image to be screened.

[0218] It is understandable that, for an image, when the predicted total score of the image is calculated by weighted sum, the sub-score of the image on a single quality assessment dimension has a limited impact on the predicted total score of the image. That is, when the quality of an image is low only in one quality assessment dimension, the sub-score of the image on this quality assessment dimension is low. However, the predicted total score of the image is affected by the sub-scores of the image on other quality assessment dimensions other than this quality assessment dimension, and may not reflect the above situation. It is understandable that in this case it is not appropriate to add the image to the image library. At this time, if the judgment is made only according to the size of the predicted total score of the image, the electronic device may add the image to the image library, resulting in low detection accuracy when subsequent detection is performed based on the images in the image library.

[0219] In some embodiments, the step of calculating the predicted total score of the current image to be screened according to the current weights of the quality evaluation dimensions represented by the predicted sub-scores and the branch networks includes:

[0220] According to each prediction sub-score and the current weight of each quality evaluation dimension represented by each branch network, the predicted total score of the current image to be screened is calculated according to the second formula.

[0221] Among them, the second formula is:

[0222] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0223] f(z) represents the predicted total score of the current image to be screened; k represents the number of quality evaluation dimensions; f(x1) represents the predicted sub-score of the current image to be screened on the first quality evaluation dimension, and α represents the current weight of the first quality evaluation dimension; f(x i ) represents the predicted sub-score of the current image to be screened on the i-th quality evaluation dimension, β represents the current weight of the i-th quality evaluation dimension; f(x k ) represents the predicted sub-score of the current image to be screened on the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0224] In an embodiment of the present application, after obtaining the predicted sub-scores of the current image to be screened in each quality evaluation dimension, for each quality evaluation dimension, the electronic device can calculate the predicted sub-score of the current image to be screened in the quality evaluation dimension, and the product of the current weight of the quality evaluation dimension (which can be called the first product). And calculate the product of the first products corresponding to each quality evaluation dimension (which can be called the second product). Then, calculate the product of the preset first parameter (i.e., w) and the second product (which can be called the third product), and use the sum of the third product and the preset second parameter (i.e., n) as the predicted total score of the current image to be screened.

[0225] The preset first parameter value (w) is greater than 0, that is, the predicted total score of the current image to be screened is positively correlated with the second product.

[0226] For example, the value range of n can be [0,1].

[0227] Correspondingly, the above method of calculating the predicted total score of the image can be called a multiplication fusion method.

[0228] In one implementation, if the quality evaluation dimensions include: image clarity, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area ratio of a specified object in the image. In the process of calculating the predicted total score of the current image to be screened by multiplication fusion, the second formula can be expressed as:

[0229] f(z)=w×(αf(x1)×βf(x2)×γf(x3)×δf(x4)×εf(x5))+n;

[0230] Among them, f(z) represents the predicted total score of the current image to be screened; f(x1) represents the predicted sub-score of the current image to be screened on the clarity of the image, α represents the current weight of the first quality evaluation dimension (image clarity); f(x2) represents the predicted sub-score of the current image to be screened on the visibility of the image, β represents the current weight of the second quality evaluation dimension (image visibility); f(x3) represents the predicted sub-score of the current image to be screened on the pitch angle of the specified object in the image, γ represents the current weight of the third quality evaluation dimension (the pitch angle of the specified object in the image); f(x4) represents the predicted sub-score of the current image to be screened on the heading angle of the specified object in the image, δ represents the current weight of the fourth quality evaluation dimension (the heading angle of the specified object in the image); f(x5) represents the predicted sub-score of the current image to be screened on the area ratio of the specified object in the image, ε represents the current weight of the fifth quality evaluation dimension (the area ratio of the specified object in the image); w is the preset first parameter, and n is the preset second parameter.

[0231] It is understandable that, for an image, when calculating the predicted total score of the image by multiplication fusion, if the sub-score of the image in a single quality evaluation dimension is low, the predicted total score of the image will also be low. That is to say, when the quality of an image is low only in one quality evaluation dimension, the sub-score of the image in the quality evaluation dimension is low, and accordingly, the predicted total score of the image is also low. In this case, the electronic device can determine that it is not appropriate to add the image to the image library based on the predicted total score of the image.

[0232] Based on the above processing, when the predicted total score of the current image to be screened is calculated based on the above multiplication fusion method, any predicted sub-score with a lower value will lead to a significant decrease in the predicted total score. In other words, the predicted total score of the current image to be screened depends on the performance of the image in each quality evaluation dimension. The defects of the image in any quality evaluation dimension will have a significant impact on the overall quality evaluation result (i.e., the predicted total score).

[0233] That is, compared with calculating the predicted total score of an image using the weighted sum method mentioned above, the predicted total score of an image is calculated using the multiplication fusion method in the embodiment of the present application, which can ensure that the images added to the image library have high quality in each quality evaluation dimension, that is, the overall quality of the image is high, and thus, when subsequent detection is performed based on the images in the image library, the accuracy of the recognition results is improved.

[0234] With respect to step S103, after obtaining the predicted total score of the current image to be screened, the electronic device may determine whether the predicted total score of the current image to be screened is less than a preset filtering threshold. For example, the preset filtering threshold may be 0.1.

[0235] If the predicted total score of the current image to be screened is less than the preset filtering threshold, it indicates that the overall quality of the current image to be screened is low. Accordingly, the current image to be screened is not suitable for subsequent detection processes. Therefore, the electronic device can determine that the screening result of the current image to be screened is not to be retained.

[0236] If the predicted total score of the current image to be screened is not less than the preset filtering threshold, it indicates that the overall quality of the current image to be screened is high. Accordingly, the current image to be screened can be used in subsequent detection processes. Therefore, the electronic device can determine that the screening result of the current image to be screened is retained.

[0237] In one implementation, the preset range may be: an adjacent interval of a specified size determined with the preset filtering threshold as the center. For example, when the preset filtering threshold is 0.1, the preset range may be [0.05, 0.2].

[0238] It can be understood that if the predicted total score of the current image to be screened belongs to the preset range, that is, the predicted total score of the current image to be screened is close to the preset filtering threshold. In other words, the current image to be screened is not an image that must be retained due to its very high image quality, nor is it an image that must not be retained due to its very poor image quality. That is, the screening result of the current image to be screened is between the edge of retention and non-retention. Correspondingly, if the current filtering ratio is lower than the target filtering ratio, it indicates that the proportion of images that are not retained as a result of the screening is small. Therefore, the intensity of the screening can be increased and the current image to be screened is not retained. If the current filtering ratio is higher than the target filtering ratio, it indicates that the proportion of images that are not retained as a result of the screening is large. Therefore, the intensity of the screening can be reduced and the current image to be screened is retained.

[0239] At this time, the electronic device can adjust the current weight of each quality evaluation dimension based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio. The specific process of adjusting the weight of each quality evaluation dimension will be described in the subsequent embodiments.

[0240] In the present application, executing the above steps S101 to S103 once can be called a scoring filtering process.

[0241] It is understandable that if the total number of images to be screened is fixed, after each execution of the process of determining the image screening result, the number of images that are not currently screened will be reduced by one.

[0242] That is, in the process of performing the scoring and filtering for the Nth time (hereinafter referred to as the Nth execution process), after the electronic device determines the screening result of the current image to be screened, it can return to the above step S101, that is, start the N+1th scoring and filtering process. It can be understood that the current image to be screened in the Nth execution process is missing from the current image to be screened in the Nth execution process, compared with the current image not screened in the Nth execution process.

[0243] Accordingly, the electronic device may perform the above-mentioned scoring and filtering process multiple times until the screening results of all the images to be screened are obtained.

[0244] For step S104, the image library represents a storage space for storing images. After obtaining the screening results of all the images to be screened, the electronic device can obtain images for adding to the current image library based on the images that are retained as a result of the screening.

[0245] For example, in the above-mentioned identity authentication scenario, after obtaining the photos uploaded by the user of all persons who have the authority to enter the designated place (i.e., all images to be screened), the electronic device can obtain the images for adding to the current image library according to the above-mentioned steps S101 to S104, and then add the part of the images for adding to the current image library to the image library.

[0246] In some embodiments, see Figure 2 , Figure 2 This is a second flow chart of the image screening method provided in an embodiment of the present application.

[0247] exist Figure 1 On the basis of this, step S104 comprises:

[0248] S1041: Selecting a currently unprocessed image from the images that are retained as a screening result as the current image to be processed.

[0249] S1042: Extract features of the current image to be processed to obtain current first image features.

[0250] S1043: Based on the matching results between the current first feature clusters and the current first image features, determine whether the current image to be processed contains a designated object.

[0251] The current first feature clusters are obtained by clustering the image features of the images included in the current first image set; the images in the current first image set do not include the specified object.

[0252] S1044: If the current image to be processed includes the designated object, determine that the current image to be processed is an image to be added to the current image library.

[0253] If there is an image that has not been processed yet, the process returns to step S1041.

[0254] In the embodiment of the present application, the electronic device may select a currently unprocessed image from the images that are retained as a result of the screening according to the order in which the screening results are determined as the current image to be processed, and process it according to steps S1042 to S1044. Alternatively, the electronic device may also randomly select one from the currently unprocessed images that are retained as a result of the screening as the current image to be processed, and process it according to steps S1042 to S1044. The specific method for determining the currently unprocessed image is not limited herein.

[0255] In the present application, executing the above steps S1041 to S1044 once can be called a modeling and filtering process. Accordingly, after each modeling and filtering process is executed, the number of currently unprocessed images will be reduced by one.

[0256] That is, during the Nth execution of modeling filtering, after the electronic device determines whether the current image to be processed contains the specified object, it can return to execute the above step S1041, that is, start the N+1th scoring filtering process. It can be understood that the current unprocessed image in the N+1th execution process lacks the current image to be processed in the Nth execution process relative to the current unprocessed image in the Nth execution process.

[0257] For steps S1042 to S1043, the electronic device can use a preset feature extraction network to extract features from the current image to be processed to obtain image features of the current image to be processed (i.e., the current first image features). For example, the preset feature extraction network can be a Transform backbone network.

[0258] The current first feature clusters are obtained by clustering the image features of the images included in the current first image set. The images in the current first image set do not include the specified object.

[0259] In one implementation, the electronic device may perform feature extraction on each image in the current first image set to obtain image features of each image in the current first image set. For example, the image features of each image in the current first image set may be expressed as: enroll ={f e1 ,f e2 ,f e3 …,f em}. Among them, f e1 represents the image features of the first image in the current first image set, f e2 represents the image features of the second image in the current first image set, f e3 represents the image features of the third image in the current first image set, and so on, f em represents the image features of the mth image in the current first image set; F enroll It represents a set of image features of all images in the current first image set that have been recorded (enrolled).

[0260] Furthermore, the electronic device may cluster the image features of each image included in the current first image set based on a preset clustering algorithm to obtain multiple feature clusters (i.e., the current first feature clusters). For example, the preset clustering algorithm may be a GCN (Graph Convolutional Networks) clustering algorithm. Accordingly, the current first feature clusters may be represented as:

[0261] GCNF enroll ={{f X1 ,f X2 ,f X3 ,…}1,{f Y1 ,f Y2 ,f Y3 ,…}2,…,{f Z1 ,f Z2 ,f Z3 ,…} n};

[0262] Among them, {f X1 ,f X2 ,f X3 …,}1 represents the first feature cluster in the current first feature clusters, f X1 represents the first image feature in the first feature cluster, f X2 represents the second image feature in the first feature cluster, f X3 represents the third image feature in the first feature cluster, and so on. The number of image features contained in the first feature cluster is not limited here. Y1 ,f Y2,f Y3 ,…}2 represents the second feature cluster in the current first feature clusters, f Y1 represents the first image feature in the second feature cluster, f Y2 represents the second image feature in the second feature cluster, f Y3 represents the third image feature in the second feature cluster, and so on. The number of image features contained in the second feature cluster is not limited here. Z1 ,f Z2 ,f Z3 ,…} n represents the nth feature cluster in the current first feature clusters, f Z1 represents the first image feature in the nth feature cluster, f Z2 represents the second image feature in the nth feature cluster, f Z3 Indicates the third image feature in the nth feature cluster, and so on. The number of image features contained in the nth feature cluster is not limited here. GCNF enroll Indicates the set of currently enrolled first feature clusters.

[0263] Accordingly, for each current first feature cluster, the electronic device can calculate the mean of the image features included in the first feature cluster as the cluster center (also called the class center) of the first feature cluster. For example, the cluster center of the jth feature cluster in each current first feature cluster can be calculated by the following formula:

[0264]

[0265] Among them, U j represents the cluster center of the jth feature cluster in the current first feature clusters, k represents the number of image features contained in the jth feature cluster, and f i Represents the i-th image feature in the j-th feature cluster.

[0266] Furthermore, in the process of performing a modeling filter, after obtaining the current first image feature, the electronic device can determine whether the current image to be processed contains a designated object based on the matching results between the current first feature clusters and the current first image feature.

[0267] In some embodiments, in the process of determining whether the current image to be processed contains the specified object based on the matching results of the current first feature clusters and the current first image feature, the electronic device may determine whether the current image to be processed contains the specified object by using the similarity between the image features. Figure 3 , Figure 3A flowchart for determining whether a current image to be processed contains a specified object is provided in an embodiment of the present application, comprising the following steps:

[0268] S301: Calculate first similarities between a current first image feature and cluster centers of current first feature clusters.

[0269] S302: Selecting a first feature cluster with the largest first similarity from among the current first feature clusters.

[0270] S303: Calculate each second similarity between the current first image feature and each image feature included in the currently selected first feature cluster.

[0271] S304: Determine whether the current image to be processed contains a designated object according to each second similarity corresponding to the current image to be processed.

[0272] In the embodiment of the present application, for each current first feature cluster, the electronic device can calculate the cosine similarity (i.e., the first similarity) between the current first image feature and the cluster center of the first feature cluster, and thus obtain the first similarities corresponding to the current first image feature. For example, the first similarities corresponding to the current first image feature can be expressed as: S Result ={cos u1α ,cos u2α ,cos u3α ,…,cos unα}.

[0273] Among them, S Result represents a set of first similarities corresponding to the current first image feature; cos u1α represents the cosine similarity between the current first image feature and the cluster center of the first feature cluster in each current first feature cluster; cos u2α Represents the cosine similarity between the current first image feature and the cluster center of the second feature cluster in each current first feature cluster; cos u3α Represents the cosine similarity between the current first image feature and the cluster center of the third feature cluster in each current first feature cluster, and so on. unα Represents the cosine similarity between the current first image feature and the cluster center of the nth feature cluster in each current first feature cluster.

[0274] Furthermore, the electronic device may select a first feature cluster (which may be referred to as the current fourth feature cluster) with the largest first similarity from the current first feature clusters. For example, the first similarity corresponding to the current fourth feature cluster may be expressed as:

[0275] costargetα =max{S Result};

[0276] Among them, max{S Result} represents the maximum value of the first similarities corresponding to the current first image feature, cos targetα Indicates the first similarity corresponding to the current fourth feature cluster.

[0277] It can be understood that since the current fourth feature cluster is: the one with the maximum first similarity value among the current first feature clusters, that is, the current fourth feature cluster is: the feature cluster most similar to the current first image feature among the current first feature clusters.

[0278] Therefore, after determining the current fourth feature cluster, the electronic device can further use the image features included in the current fourth feature cluster to determine whether the current image to be processed includes the designated object.

[0279] Accordingly, the electronic device can use the DAM (Difference Alignment Metric) algorithm to obtain the second similarities corresponding to the current image to be processed. That is, for each image feature included in the current fourth feature cluster, the electronic device can calculate the cosine similarity (i.e., the second similarity) between the current first image feature and the image feature, and determine whether the current image to be processed contains the specified object based on the second similarities corresponding to the current image to be processed.

[0280] In some embodiments, step S304 includes:

[0281] Step 1: Determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed.

[0282] Step 2: If the first preset number of largest second similarities are all greater than the first preset similarity threshold, it is determined that the current image to be processed does not contain the designated object.

[0283] Step 3: Otherwise, determine whether the current image to be processed contains the specified object.

[0284] In the embodiment of the present application, after obtaining the second similarities corresponding to the current image to be processed, the electronic device may determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed.

[0285] Furthermore, since the image features in the current fourth feature cluster are: image features that do not contain the specified object, if the first preset number of largest second similarities are all greater than the first preset similarity threshold, it indicates that the image features of the current image to be processed are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the current image to be processed does not contain the specified object.

[0286] Otherwise, if the first preset number of largest second similarities are not all greater than the first preset similarity threshold, it indicates that the image features of the current image to be processed are not similar to the image features of the image not containing the specified object. Therefore, it can be determined that the current image to be processed contains the specified object.

[0287] In one implementation, in order to facilitate the comparison of similarity, after calculating the cosine similarities between the current first image feature and the image features included in the current fourth feature cluster, the cosine similarities may be normalized, that is, the values ​​of the cosine similarities are mapped to [0, 1] to obtain the second similarities between the current first image feature and the image features included in the current fourth feature cluster.

[0288] In this way, after determining the feature cluster closest to the current first image feature (i.e., the current fourth feature cluster), each image feature contained in the current fourth feature cluster can be further used to determine whether the current image to be processed contains the specified object. That is, based on the first preset number of largest second similarities, determining whether the current image to be processed contains the specified object can reduce the amount of calculation and improve the efficiency of image screening.

[0289] Based on the above processing, the current first image feature is first compared with the cluster center of each feature cluster to determine the feature cluster closest to the current first image feature (i.e., the current fourth feature cluster), and then the current first image feature is compared with each image feature included in the current fourth feature cluster. In this way, compared with the method of directly comparing with each image feature, the amount of calculation can be reduced, and the image features of the recorded image can be used to quickly identify the current image to be processed.

[0290] With respect to step S1044 , if the current image to be processed includes the designated object, it is determined that the current image to be processed is an image to be added to the current image library, and the process returns to step S1041 .

[0291] In some embodiments, the method further comprises:

[0292] When it is determined that the current image to be processed does not contain the designated object, the current image to be processed is added to the current first image set.

[0293] In an embodiment of the present application, if it is determined that the current image to be processed does not contain the specified object, the electronic device can add the current image to be processed to the current first image set. In the subsequent process of executing the above-mentioned modeling and filtering, each feature cluster can be determined based on the new current first image set.

[0294] In this way, dynamic updating of the first image set can be achieved, that is, images that do not contain the specified object (i.e., negative samples) recorded in the first image set can be continuously updated during the image screening process. In other words, images that do not contain the specified object obtained during the image screening process can be retained and involved in subsequent screening processes. In this way, the accuracy of subsequent image screening can be improved, and continuous optimization of screening accuracy can be achieved.

[0295] Based on the above processing, for each image to be added to the image library, the image can first be determined as the current image to be screened, and the above-mentioned scoring and filtering process can be performed. In the case where the screening result of the image is retained, the image is determined as the current image to be processed, and the above-mentioned modeling and filtering process is performed. In the case where it is determined that the current image to be processed contains a specified object, the image is determined to be an image for adding to the current image library. In this way, the image can be combined with the above-mentioned scoring filtering and modeling filtering methods to perform multi-level filtering on the image, which can also achieve the accuracy of image screening and improve the quality of images stored in the image library.

[0296] In some embodiments, for each image to be added to the image library, the above modeling and filtering process may be performed first, and when it is determined that the current image to be processed contains a specified object, the above scoring and filtering process may be performed. Figure 4 , Figure 4 This is a third flow chart of the image screening method provided in the embodiment of the present application. Figure 1 On the basis of, before step S102, the method further includes:

[0297] S105: Extract features of the current image to be screened to obtain current second image features.

[0298] S106: Based on the matching results between the current second feature clusters and the current second image features, determine whether the current image to be screened contains the designated object.

[0299] The current second feature clusters are obtained by clustering the image features of the images included in the current second image set; the images in the current second image set do not include the specified object.

[0300] Step S102 includes:

[0301] S1021: When the current image to be screened contains the specified object, the current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and the predicted total score of the current image to be screened is calculated based on the current weights of each predicted sub-score and each quality assessment dimension represented by each branch network.

[0302] In the embodiment of the present application, the process of extracting features of the current image to be screened to obtain the current second image features can refer to the process of obtaining the current first image features in the above step S1042, and will not be repeated here.

[0303] The process of obtaining the current second feature clusters can refer to the process of obtaining the current first feature clusters in the above embodiment, which will not be described in detail here. The description of the current second image set can refer to the description of the current first image set in the above embodiment, which will not be described in detail here.

[0304] The process of determining whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features can be referred to the above-mentioned step S1043, which is the process of determining whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features, and will not be repeated here.

[0305] In some embodiments, step S106 includes:

[0306] Step 1: Calculate the third similarities between the current second image feature and the cluster centers of the current second feature clusters.

[0307] Step 2: From the current second feature clusters, select the second feature cluster with the largest third similarity.

[0308] Step three: Calculate each fourth similarity between the current second image feature and each image feature included in the currently selected second feature cluster.

[0309] Step 4: Determine whether the current image to be screened contains the specified object according to each fourth similarity corresponding to the current image to be screened.

[0310] In the embodiment of the present application, the process of calculating the third similarities corresponding to the current second image feature can refer to the process of calculating the first similarities corresponding to the current first image feature in the above step S301, and will not be repeated here.

[0311] Furthermore, the electronic device may select a second feature cluster (which may be referred to as the current fifth feature cluster) with the largest corresponding third similarity from the current second feature clusters.

[0312] It can be understood that since the current fifth feature cluster is: the one with the maximum third similarity value among the current second feature clusters, that is, the current fifth feature cluster is: the feature cluster most similar to the current second image feature among the current second feature clusters.

[0313] Therefore, after determining the current fifth feature cluster, the electronic device can further use the image features included in the current fifth feature cluster to determine whether the current image to be processed contains the specified object. That is, the fourth similarities between the current second image feature and the image features included in the current fifth feature cluster (i.e., the currently selected second feature cluster) are calculated.

[0314] The process of determining whether the current image to be filtered contains the specified object according to the fourth similarities corresponding to the current image to be filtered can refer to the process of determining whether the current image to be processed contains the specified object according to the second similarities corresponding to the current image to be processed in the above step S304, which will not be repeated here.

[0315] In one implementation, the above step 4 includes:

[0316] Determine the fourth similarities with the second preset number of largest ones among the fourth similarities corresponding to the current image to be screened. If the second preset number of largest fourth similarities are all greater than the second preset similarity threshold, determine that the current image to be screened does not contain the specified object; otherwise, determine that the current image to be screened contains the specified object.

[0317] In the embodiment of the present application, the process of determining the second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened can refer to the process of determining the first preset number of largest second similarities among the second similarities corresponding to the current image to be processed in the above step 1.

[0318] Furthermore, since the image features in the current fifth feature cluster are: image features that do not contain the specified object, if the second preset number of fourth similarities are all greater than the second preset similarity threshold, it indicates that the image features of the current image to be screened are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the current image to be screened does not contain the specified object.

[0319] Otherwise, if the second preset number of fourth similarities are not all greater than the second preset similarity threshold, it indicates that the image feature of the current image to be screened is not similar to the image feature of the image not containing the specified object. Therefore, it can be determined that the current image to be screened contains the specified object.

[0320] In this way, after determining the feature cluster closest to the current second image feature (i.e., the current fifth feature cluster), the image features contained in the current fifth feature cluster can be further used to determine whether the current image to be processed contains the specified object. That is, based on the first second preset number of fourth similarities with larger values, determining whether the current image to be screened contains the specified object can reduce the amount of calculation and improve the efficiency of image screening.

[0321] Based on the above processing, the current second image feature is first compared with the cluster center of each feature cluster to determine the feature cluster (i.e., the current fifth feature cluster) that the current second image feature is closest to, and then the current second image feature is compared with each image feature included in the current fifth feature cluster. In this way, compared with the method of directly comparing with each image feature, the amount of calculation can be reduced, and the image features of the recorded image can be used to quickly identify the current image to be screened.

[0322] Accordingly, if the current image to be processed contains the specified object, the scoring filtering process can be performed. That is, when the current image to be screened contains the specified object, the current image to be screened is input into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model. Furthermore, the electronic device can calculate the predicted total score of the current image to be screened based on each predicted sub-score and the current weight of each quality assessment dimension represented by each branch network.

[0323] In some embodiments, the method further comprises:

[0324] When it is determined that the current image to be screened does not contain the specified object, the current image to be screened is added to the current second image set, and the process returns to step S101.

[0325] In an embodiment of the present application, if it is determined that the current image to be screened does not contain the specified object, the electronic device can add the current image to be screened to the current second image set. In the subsequent process of executing the above-mentioned modeling and filtering, each feature cluster can be determined based on the new current second image set.

[0326] In this way, dynamic updating of the second image set can be achieved, that is, images that do not contain specified objects (i.e., negative samples) recorded in the second image set can be continuously updated during the image screening process. In other words, images that do not contain specified objects obtained during the image screening process can be retained and involved in subsequent screening processes. In this way, the accuracy of subsequent image screening can be improved, and continuous optimization of screening accuracy can be achieved.

[0327] Based on the above processing, for each image to be added to the image library, the image can first be determined as the current image to be screened, and the above modeling and filtering process can be performed. When it is determined that the current image to be screened contains the specified object, the above scoring and filtering process is performed. When the screening result of the image is retained, the image is determined to be an image to be added to the current image library. In this way, the image can be combined with the above scoring and modeling filtering methods to perform multi-level filtering on the image, which can achieve the accuracy of image screening and improve the quality of the images stored in the image library.

[0328] In some embodiments, the electronic device can adjust the current weight of each quality evaluation dimension based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio. Figure 5 , Figure 5 A flowchart for adjusting the current weight of each quality evaluation dimension provided in an embodiment of the present application includes the following steps:

[0329] S501: Get the current status.

[0330] The current status includes: the current filtering ratio, the number of images for which the filtering results have been determined, and the current weight of each quality evaluation dimension.

[0331] S502: Determine the expected benefit when taking each alternative action in the current state as the current first expectation.

[0332] Among them, an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension.

[0333] S503: Select the candidate action corresponding to the current largest first expectation as the action to be taken currently.

[0334] S504: Based on the current action to be taken, adjust the current weight of each quality evaluation dimension.

[0335] In the embodiment of the present application, the current state includes: the current filtering ratio, the number of images for which the screening results have been determined, and the current weight of each quality evaluation dimension.

[0336] For example, when the above-mentioned quality evaluation dimensions include: image clarity, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area ratio of a specified object in the image, the current state can be expressed as: S = (n, p, α, β, γ, δ, ε).

[0337] Among them, S represents the current state, n represents the number of images for which the screening results have been determined, p represents the current filtering ratio, α represents the current weight of the image clarity, β represents the current weight of the image visibility, γ represents the current weight of the pitch angle of the specified object in the image, δ represents the current weight of the heading angle of the specified object in the image, and ε represents the current weight of the area ratio of the specified object in the image.

[0338] The current filtering ratio is the ratio of the number of images that are not retained in the screening results to the number of images that have been determined to be filtered. For example, the number of images that are not retained in the screening results can be recorded as n1, and the number of images that are retained in the screening results can be recorded as n2. Correspondingly, the number of images that have been determined to be filtered (n) is the sum of n1 and n2. The current filtering ratio p can be expressed as:

[0339] An alternative action includes: the adjustment amount of the weight of each quality evaluation dimension. For example, if the number of alternative actions is m, the first alternative action can be recorded as A1, the second alternative action can be recorded as A2, and so on, the mth alternative action can be recorded as An.

[0340] Accordingly, the electronic device determines the expectation of gain when taking each alternative action in the current state as the current first expectation. For example, the current first expectation includes: Q(S, A1), Q(S, A2), ..., Q(S, Am). Among them, Q(S, A1) represents the expectation of gain when taking the first alternative action (A1) in the current state, Q(S, A2) represents the expectation of gain when taking the second alternative action (A2) in the current state, and so on, Q(S, Am) represents the expectation of gain when taking the nth alternative action (Am) in the current state.

[0341] After obtaining the current first expectation, the electronic device can select the candidate action corresponding to the current maximum first expectation as the action to be taken currently. The action to be taken currently can be recorded as A, and accordingly, the action to be taken currently can be expressed as: A = (Δα, Δβ, Δγ, Δδ, Δε).

[0342] Among them, Δα represents the adjustment amount of the weight of the image clarity, Δβ represents the adjustment amount of the weight of the image visibility, Δγ represents the adjustment amount of the weight of the pitch angle of the specified object in the image, Δδ represents the adjustment amount of the weight of the heading angle of the specified object in the image, and Δε represents the adjustment amount of the weight of the area ratio of the specified object in the image.

[0343] Furthermore, the electronic device can adjust the current weights of each quality evaluation dimension based on the action currently required. That is, the sum of the current weight (α) of the image clarity and the adjustment amount (Δα) of the weight of the image clarity, the sum of the current weight (β) of the image visibility and the adjustment amount (Δβ) of the weight of the image visibility, the sum of the current weight (γ) of the pitch angle of the specified object in the image and the adjustment amount (Δγ) of the weight of the pitch angle of the specified object in the image, the sum of the current weight (δ) of the heading angle of the specified object in the image and the adjustment amount (Δδ) of the weight of the heading angle of the specified object in the image, the sum of the current weight (ε) of the area ratio of the specified object in the image and the adjustment amount (Δε) of the weight of the area ratio of the specified object in the image, and thus the current weights of each quality evaluation dimension are updated.

[0344] In the process of executing the above steps S102 to S103 for the current image to be screened, after the electronic device determines the screening result of the current image to be screened, p (current filtering ratio) in the current state (S) will change. Correspondingly, if p in the current state (S) does not reach the target filtering ratio, the electronic device can update the current weights of each quality evaluation dimension according to S501 to S504 above. If p in the current state (S) reaches the target filtering ratio, it is considered that the current state reaches the target state (which can be called Target S). At this time, there is no need to update the current weights of each quality evaluation dimension.

[0345] It is understandable that after returning to step S101 and determining the next current image to be screened, the electronic device can re-determine whether to update the current weights of each quality evaluation dimension according to the current state until the screening results of all images to be screened are determined.

[0346] Based on the above processing, in the process of screening images, the target filtering ratio and the current filtering ratio can be further combined to dynamically adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm. In the subsequent image screening process, the predicted total score of the image is determined based on the updated weights of each quality evaluation dimension, and then the image screening result is determined. In other words, at each stage of screening all images to be screened, the current filtering ratio can be continuously adjusted to the target filtering ratio, thereby improving the flexibility of image screening and meeting the personalized needs of users.

[0347] In addition, it is understandable that the current weights of each quality evaluation dimension are adjusted only when the predicted total score of the current image to be screened falls within the preset range. In this way, it can be ensured that the predicted total score of images with a higher predicted total score (i.e., better image quality) is not affected by the weight adjustment, and it can also be ensured that the correctly retained images are not affected by the weight adjustment. Moreover, the predicted total score of images with a lower predicted total score (i.e., poorer image quality) is not affected by the weight adjustment, and it can also be ensured that the correctly screened images are not affected by the weight adjustment.

[0348] In some embodiments, the method further comprises:

[0349] After determining the action currently required to be taken, the current first expectation is updated based on the first formula.

[0350] Among them, the first formula is:

[0351] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0352]

[0353] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken at present; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after the screening result of the current image to be screened is obtained, maxQ(S′,A′) represents the maximum value of the second expectation of the benefit obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value in the second expectation;

[0354] st represents the constraint condition, R(S,A) represents: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c Represents the preset filtering threshold, R min Indicates the minimum value in the preset range, R max Indicates the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other situations.

[0355] In an embodiment of the present application, after determining the action currently required to be taken, the electronic device can calculate the reward R according to the incentive function R(S, A) in the above first formula, and update the current first expectation.

[0356] Among them, if the predicted total score f(z) of the currently to-be-screened image satisfies f(z) < R c and (and) R min < f(z) < R max , and the current filtering ratio p satisfies p < M, then the value of the reward R is 1; if the predicted total score f(z) of the currently to-be-screened image satisfies f(z) > R max , or the current filtering ratio p satisfies p ≥ M, then the value of the reward R is -1; otherwise indicates that the predicted total score f(z) of the currently to-be-screened image and the current filtering ratio p do not satisfy any of the above conditions. At this time, the value of the reward R is 0.

[0357] Based on the above processing, in the process of updating the current first expectation, the electronic device can set a reward function according to the relationship between the current filtering ratio and the target filtering ratio. That is to say, if the current filtering ratio is less than the target filtering ratio, a positive reward is given, that is, the value of R is 1; if the current filtering ratio is not less than the target filtering ratio, a negative reward is given, that is, the value of R is -1.

[0358] In some embodiments, refer to Figure 6 , Figure 6 is a flowchart of a training quality evaluation model provided by an embodiment of the present application. The quality evaluation model is trained through the following steps:

[0359] S601: Obtain a sample image including a specified object.

[0360] Among them, the label of the sample image includes: the sample sub-scores of the sample image in each quality evaluation dimension, and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values of the sample sub-scores of the sample image and the weights of each quality evaluation dimension.

[0361] S602: For each sample image, input the sample image into the quality evaluation model with an initial structure, obtain the training sub-scores output by each branch network in the quality evaluation model with the initial structure, and calculate the training total score of the sample image according to the training sub-scores and the initial values of the weights of each quality evaluation dimension represented by each branch network.

[0362] S603: Calculate a loss value based on the training sub-scores and training total score of the sample image, and the sample sub-scores and sample total score of the sample image.

[0363] S604: Adjust the model parameters of the quality evaluation model with the initial structure based on the obtained loss value until convergence is achieved.

[0364] In the embodiment of the present application, the sample image may be an image in a public image collection.

[0365] It is understandable that the designated object in the image screening process is consistent with the type of the designated object in the sample image, that is, if the designated object in the image to be screened is a human face, then the designated object in the sample image is also a human face.

[0366] The quality evaluation model is composed of a backbone network and multiple fully-connected layers (FC, fully-connected) (i.e., branch networks in this application). Among them, the backbone network can be a convolutional neural network (CNN), and one fully-connected layer corresponds to one quality evaluation dimension.

[0367] Correspondingly, after obtaining an image, the quality assessment model can input the image into the backbone network (CNN), and input the data output by the backbone network into each fully connected layer, and each fully connected layer can process the input data. Furthermore, for each fully connected layer, the output result of the fully connected layer can be processed based on the activation function corresponding to the fully connected layer to obtain the sub-score of the image on the quality assessment dimension represented by the fully connected layer.

[0368] Correspondingly, for each sample image, the sample image is input into the quality evaluation model of the initial structure to obtain the training sub-score output by each branch network in the quality evaluation model of the initial structure.

[0369] Among them, the process of calculating the total training score of the sample image according to the initial values ​​of the weights of each training sub-score and each quality evaluation dimension represented by each branch network can refer to the process of calculating the predicted total score of the current image to be screened according to the current weights of each prediction sub-score and each quality evaluation dimension represented by each branch network in the above-mentioned step S102, which will not be repeated here.

[0370] The label of the sample image includes: the sample sub-score of the sample image in each quality evaluation dimension, and the sample total score of the sample image.

[0371] The range of sample sub-scores of the sample images in each quality evaluation dimension is preset according to actual needs. For example, the range of each sample sub-score may be [0, 100].

[0372] In one implementation, during the process of marking a sample image, the sample sub-score of the sample image in each quality evaluation dimension can be determined by comparing it with another reference sample image in each quality evaluation dimension.

[0373] For example, if the quality evaluation dimensions include: image clarity, image visibility, and the area ratio of the specified object in the image. And, the sample sub-score of the reference sample image on image clarity is 50, the sample sub-score of the reference sample image on image visibility is 50, and the sample sub-score of the reference sample image on the area ratio of the specified object in the image is 50. If a sample image (which can be called sample image 1) has higher clarity than the reference sample image, the sample sub-score of the sample image 1 on image clarity can be set to be greater than 50 (such as 60); if the visibility of the sample image 1 is also higher than that of the reference sample image, the sample sub-score of the sample image 1 on image visibility can be set to be greater than 50 (such as 70); if the area ratio of the specified object in the sample image 1 is smaller than that of the reference sample image, the sample sub-score of the sample image 1 on image visibility can be set to be less than 50 (such as 40).

[0374] Accordingly, after determining the sample sub-scores of a sample image in each quality evaluation dimension, the total sample score of the sample image can be calculated according to each sample sub-score and the initial value of the weight of each quality evaluation dimension represented by each branch network. For example, the multiplication fusion method in the above embodiment can be referred to, which will not be described in detail here.

[0375] Accordingly, for a quality evaluation dimension, the electronic device can calculate the loss value of the sample image in the quality evaluation dimension based on the difference between the training sub-score of the sample image in the quality evaluation dimension and the sample sub-score of the sample image in the quality evaluation dimension. Further, the sub-score loss value of the sample image in each quality evaluation dimension is obtained.

[0376] Furthermore, for a sample image, the electronic device may calculate a total score loss value based on a difference between a total training score of the sample image and a total sample score of the sample image.

[0377] Furthermore, the electronic device can calculate the weighted sum of the sub-score loss values ​​and the total score loss value corresponding to the sample image, and adjust the model parameters of the quality evaluation model of the initial structure based on the weighted sum until convergence is reached.

[0378] Based on the above processing, the image screening method provided by the present application can directly evaluate the quality of the image through the quality evaluation model. That is, in the process of evaluating the quality of the image, the user does not need to have another undamaged reference image as the basis for evaluation, that is, in the present application, the method for evaluating the image quality is No-Reference Image Quality Assessment (NR-IQA).

[0379] Based on the same inventive concept, an embodiment of the present application provides an image recognition method, which can be applied to electronic devices.

[0380] See also Figure 7 , Figure 7 The first flow chart of the image recognition method provided in the embodiment of the present application comprises the following steps:

[0381] S701: Acquire an image to be recognized.

[0382] S702: Extract features of the image to be identified to obtain features of the image to be identified.

[0383] S703: Obtaining a recognition result of the image to be recognized based on a matching result between the features of the image to be recognized and the image features of the images in the current image library.

[0384] Among them, the images in the current image library are obtained based on the image screening method provided in this application.

[0385] Based on the above processing, an image that is not currently screened can be selected from all the images to be screened as the current image to be screened, and the current image to be screened can be input into the pre-trained quality evaluation model. The quality evaluation model includes multiple branch networks, and one branch network represents a quality evaluation dimension. Accordingly, the quality evaluation model can output the sub-scores (i.e., predicted sub-scores) of the current image to be screened on each quality evaluation dimension. Furthermore, the predicted total score of the current image to be screened can be calculated based on the current weights of each predicted sub-score and each quality evaluation dimension. According to the predicted total score of the current image to be screened, the screening result (retain or not) of the current image to be screened is determined. Accordingly, the screening result of each image in all the images to be screened can be obtained according to the above steps. In this way, automatic scoring and screening for each image to be screened can be achieved through the pre-trained quality evaluation model, without the need to manually screen the images to be screened, thereby improving the screening efficiency.

[0386] After obtaining the screening results of each image to be screened, since the screening results are determined based on the predicted total score of the image, the predicted total score of the image can reflect the overall quality of the image in multiple quality evaluation dimensions. Therefore, based on the images that are retained as a result of the screening, images for adding to the current image library are obtained, which can ensure that images with higher quality are added to the current image library for subsequent processing. Furthermore, the image recognition method provided in the embodiment of the present application can use the image features of the images in the image library to detect the images to be identified and obtain the recognition results of the images to be identified. In this way, the accuracy of the recognition results of the images to be identified can be guaranteed.

[0387] With respect to step S701 to step S702, the image to be recognized acquired by the electronic device represents: an image for which detection of a designated object contained in the image is required.

[0388] For example, in the above-mentioned identity authentication scenario, the image to be identified may be: an image captured by a gate at the entrance of a designated place and containing a person who currently needs to enter the designated place.

[0389] Furthermore, the electronic device can use a preset feature extraction network to extract features from the image to be identified, and obtain image features of the image to be identified (ie, features of the image to be identified). For example, the preset feature extraction network can be a Transform backbone network.

[0390] With respect to step S703 , based on the matching result between the features of the image to be identified and the image features of the images in the current image library, the recognition result of the image to be identified is obtained.

[0391] In one implementation, the electronic device can calculate the similarity between the features of the image to be identified and the image features of each image in the current image library. Then, from the images in the current image library, the image with the greatest similarity is determined as the target image. If the similarity corresponding to the target image is greater than a preset similarity threshold, it is determined that the identity of the specified object in the image to be identified is consistent with the identity of the specified object in the target image. That is, the recognition result of the image to be identified indicates: the identity of the specified object in the image to be identified.

[0392] In some embodiments, see Figure 8 , Figure 8 A second flow chart of the image recognition method provided in the embodiment of the present application, in Figure 7 On the basis of, before step S703, the method further includes:

[0393] S704: Calculate the fifth similarities between the image feature to be identified and the cluster centers of the current third feature clusters.

[0394] The current third feature clusters are obtained by clustering the image features of the images included in the current third image set; the images in the current third image set do not include the specified object.

[0395] S705: Selecting the third feature cluster with the largest fifth similarity from the current third feature clusters.

[0396] S706: Calculate each sixth similarity between the image feature to be identified and each image feature included in the currently selected third feature cluster.

[0397] S707: Determine whether the image to be identified contains a designated object according to each sixth similarity corresponding to the image to be identified.

[0398] Step S703 includes:

[0399] S7031: When the image to be identified contains a specified object, a recognition result of the image to be identified is obtained based on a matching result between features of the image to be identified and image features of images in the current image library.

[0400] In the embodiment of the present application, the process of obtaining the current third feature clusters can refer to the process of obtaining the current first feature clusters in the above embodiment, and will not be described in detail here.

[0401] The current third image set may be the first image set or the second image set in the present application, or may be a new image set generated during the image recognition process.

[0402] The process of calculating each fifth similarity corresponding to the detected image feature may refer to the process of calculating each first similarity corresponding to the current first image feature in the above step S301, and will not be described in detail here.

[0403] Furthermore, the electronic device may select a third feature cluster (which may be referred to as the current sixth feature cluster) having the largest corresponding fifth similarity from among the current third feature clusters.

[0404] It can be understood that since the current sixth feature cluster is: the one with the fifth maximum similarity value among the current third feature clusters, that is, the current sixth feature cluster is: the feature cluster most similar to the image feature to be identified among the current third feature clusters.

[0405] Therefore, after determining the current sixth feature cluster, the electronic device can further use the image features included in the current sixth feature cluster to determine whether the image to be identified contains the specified object. That is, the sixth similarities between the image features to be identified and the image features included in the currently selected third feature cluster (i.e., the current sixth feature cluster) are calculated.

[0406] The process of determining whether the image to be identified contains the specified object according to the sixth similarities corresponding to the image to be identified can refer to the process of determining whether the current image to be processed contains the specified object according to the second similarities corresponding to the current image to be processed in the above step S304, and will not be repeated here.

[0407] In some embodiments, step S707 includes:

[0408] Determine the sixth similarities with the largest third preset number among the sixth similarities corresponding to the image to be identified; if the third preset number of largest sixth similarities are all greater than the third preset similarity threshold, determine that the image to be identified does not contain the specified object; otherwise, determine that the image to be identified contains the specified object.

[0409] In the embodiment of the present application, the process of determining the third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified can refer to the process of determining the first preset number of largest second similarities among the second similarities corresponding to the current image to be processed in the above step 1.

[0410] Furthermore, since the image features in the current sixth feature cluster are: image features that do not contain the specified object, if the third preset number of sixth similarities are all greater than the third preset similarity threshold, it indicates that the image features of the image to be identified are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the image to be identified does not contain the specified object.

[0411] If the third preset number of sixth similarities are not all greater than the third preset similarity threshold, it indicates that the image features of the image to be identified are not similar to the image features of the image not containing the specified object. Therefore, it can be determined that the image to be identified contains the specified object.

[0412] In this way, determining whether the image to be identified contains a specified object based on the first third preset number of sixth similarities with larger values ​​can reduce the amount of calculation and improve the efficiency of image screening.

[0413] Correspondingly, when the image to be identified contains a specified object, the electronic device can obtain the identification result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0414] In some embodiments, the method further comprises:

[0415] When the image to be identified does not contain the specified object, the image to be identified is added to the current third image set.

[0416] In an embodiment of the present application, if it is determined that the image to be identified does not contain the specified object, the electronic device can add the image to be identified to the current third image set. In the subsequent process of executing the above-mentioned modeling and filtering, each feature cluster can be determined based on the new current image set.

[0417] In this way, dynamic updating of the image set can be achieved, that is, the images that do not contain the specified objects (i.e., negative samples) recorded in the image set can be continuously updated during the image screening process. In other words, the images that do not contain the specified objects obtained during the image recognition process can be retained and involved in the subsequent screening process. In this way, the accuracy of subsequent image recognition can be improved.

[0418] See also Fig. 9 , Fig. 9 A schematic diagram of a process for detecting an image provided in an embodiment of the present application. Fig. 9 In the process, after obtaining a captured image (i.e., the image to be identified), feature extraction can be performed on the image to be identified to obtain image features of the image to be identified. Furthermore, the electronic device can determine whether the image to be identified contains a specified object (i.e., perform feature filtering) based on the image features of the image to be identified and the image features in the feature set.

[0419] If the image to be identified contains the specified object, the image features of the image to be identified can be matched with the image features of the images in the image library (i.e., feature comparison can be performed) to obtain the recognition result of the image to be identified, and the detection process ends.

[0420] If the image to be identified does not contain the specified object, the image features of the image to be identified can be added to the feature set (ie, the feature set is added), and the detection process ends.

[0421] In the above detection process, the dotted line between adding the image library and the feature comparison indicates that the images used in the feature comparison process are images pre-added to the image library based on the image screening method provided in this application. Accordingly, in the process of image screening, for each image in the library (i.e., the image currently to be screened), the image in the library can be input into a pre-trained quality evaluation model to obtain the predicted sub-scores of the image in the library, and the predicted total score of the image in the library is calculated based on the current weights of the predicted sub-scores and the quality evaluation dimensions (i.e., the image in the library is scored).

[0422] Furthermore, the stored image may be subjected to threshold filtering. If the predicted total score of the stored image is less than the preset filtering threshold, the screening result of the stored image is determined to be not retained, and the image screening process ends.

[0423] If the predicted total score of the stored image is not less than the preset filtering threshold, the screening result of the stored image is determined to be retained. Then, feature extraction can be performed on the stored image to obtain image features of the stored image. The electronic device can determine whether the stored image contains a specified object (i.e., perform feature filtering) based on the image features of the stored image and the image features in the feature set.

[0424] If the stored image contains the specified object, the stored image can be added to the image library (ie, the image library is added), and the image screening process ends.

[0425] If the stored image does not contain the specified object, the image features of the stored image can be added to the feature set (ie, the feature set is added), and the image screening process ends.

[0426] Based on the same inventive concept, the present application embodiment provides an image screening device. Fig.10 , Fig.10 A structural diagram of an image screening device provided in an embodiment of the present application, the device comprising:

[0427] The first image selection module 1001 is used to select an image that is not currently being screened from all images to be screened as the current image to be screened;

[0428] The first scoring module 1002 is used to input the current image to be screened into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the current image to be screened according to each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein the quality assessment model is obtained by training based on the sample sub-scores of the sample images containing the specified objects on each quality assessment dimension and the sample total score of the sample images; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality assessment dimension;

[0429] The screening result determination module 1003 is used to adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio if the predicted total score of the current image to be screened belongs to the preset range, and determine that the screening result of the current image to be screened is retained when the predicted total score of the current image to be screened is not less than the preset filtering threshold, and determine that the screening result of the current image to be screened is not retained when the predicted total score of the current image to be screened is less than the preset filtering threshold; wherein the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose screening results are not retained to the number of images whose screening results have been determined at present; return to trigger the first image selection module 1001 until the screening results of all images to be screened are obtained;

[0430] The image adding module 1004 is used to obtain images to be added to the current image library based on the images that are retained as a result of the screening.

[0431] In some embodiments, the image adding module 1004 includes:

[0432] A first image selection submodule is used to select a currently unprocessed image from the images that are retained in the screening results as a current image to be processed;

[0433] A first feature extraction submodule is used to extract features of the current image to be processed to obtain a current first image feature;

[0434] A first object determination submodule is used to determine whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features; wherein the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; and the images in the current first image set do not contain the specified object;

[0435] The first image adding submodule is used to determine that the current image to be processed is an image to be added to the current image library if the current image to be processed contains the designated object, and trigger the first image selecting submodule.

[0436] In some embodiments, the first object determination submodule includes:

[0437] A first similarity calculation unit, used for calculating first similarities between a current first image feature and a cluster center of each current first feature cluster;

[0438] A first feature cluster selection unit, configured to select a first feature cluster with the largest first similarity from among the current first feature clusters;

[0439] A second similarity calculation unit, used for calculating each second similarity between the current first image feature and each image feature included in the currently selected first feature cluster;

[0440] The first object determination unit is used to determine whether the current image to be processed contains a designated object according to each second similarity corresponding to the current image to be processed.

[0441] In some embodiments, the first object determination unit is specifically configured to:

[0442] Determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed;

[0443] If the first preset number of largest second similarities are all greater than the first preset similarity threshold, it is determined that the current image to be processed does not contain the designated object; otherwise, it is determined that the current image to be processed contains the designated object.

[0444] In some embodiments, the apparatus further comprises:

[0445] The first image set updating module is used to add the current image to be processed to the current first image set when it is determined that the current image to be processed does not contain the specified object.

[0446] In some embodiments, the apparatus further comprises:

[0447] A second feature extraction module is used to extract features from the current image to be screened to obtain a current second image feature before inputting the current image to be screened into a pre-trained quality assessment model to obtain a predicted sub-score output by each branch network in the quality assessment model;

[0448] A first object determination module is used to determine whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features; wherein the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; and the images in the current second image set do not contain the specified object;

[0449] The first scoring module 1002 is specifically used for:

[0450] In the case that the current image to be screened contains the designated object, the current image to be screened is input into a pre-trained quality assessment model to obtain a prediction sub-score output by each branch network in the quality assessment model.

[0451] In some embodiments, the first object determination module includes:

[0452] A first similarity calculation submodule, used for calculating third similarities between the current second image feature and the cluster center of each current second feature cluster;

[0453] A first feature cluster selection submodule is used to select a second feature cluster with the largest third similarity from the current second feature clusters;

[0454] A second similarity calculation submodule, used for calculating each fourth similarity between the current second image feature and each image feature included in the currently selected second feature cluster;

[0455] The second object determination submodule is used to determine whether the current image to be screened contains a specified object according to each fourth similarity corresponding to the current image to be screened.

[0456] In some embodiments, the second object determination submodule is specifically configured to:

[0457] Determine a second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened;

[0458] If the second preset number of largest fourth similarities are all greater than the second preset similarity threshold, it is determined that the current image to be screened does not contain the specified object; otherwise, it is determined that the current image to be screened contains the specified object.

[0459] In some embodiments, the apparatus further comprises:

[0460] The second image set updating module is used to add the current image to be screened to the current second image set and trigger the first image selection module 1001 when it is determined that the current image to be screened does not contain the specified object.

[0461] In some embodiments, the screening result determination module 1003 is specifically used to:

[0462] Get the current status; the current status includes: the current filtering ratio, the number of images for which the filtering results have been determined, and the current weight of each quality evaluation dimension;

[0463] Determine the expected benefit when taking each alternative action in the current state as the current first expectation; wherein an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension;

[0464] Select the alternative action corresponding to the current largest first expectation as the action to be taken currently;

[0465] Based on the current actions that need to be taken, the current weights of each quality evaluation dimension are adjusted.

[0466] In some embodiments, the apparatus further comprises:

[0467] An expectation updating module, configured to update a current first expectation based on a first formula after determining the action currently required to be taken;

[0468] Among them, the first formula is:

[0469] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0470]

[0471] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken at present; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after the screening result of the current image to be screened is obtained, maxQ(S′,A′) represents the maximum value of the second expectation of the benefit obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value in the second expectation;

[0472] st represents the constraint condition, R(S,A) represents: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c represents the preset filtering threshold, R min Indicates the minimum value in the preset range, R max represents the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other situations.

[0473] In some embodiments, the quality assessment model is trained by the following steps:

[0474] Acquire a sample image containing a specified object; wherein the label of the sample image comprises: a sample sub-score of the sample image in each quality evaluation dimension, and a sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and weights of each quality evaluation dimension of the sample image;

[0475] For each sample image, the sample image is input into the quality evaluation model of the initial structure to obtain the training sub-scores output by each branch network in the quality evaluation model of the initial structure, and the total training score of the sample image is calculated according to the initial values ​​of the weights of each training sub-score and each quality evaluation dimension represented by each branch network;

[0476] Based on each training sub-score and the total training score of the sample image, as well as each sample sub-score and the total sample score of the sample image, a loss value is calculated;

[0477] Based on the obtained loss value, the model parameters of the quality evaluation model of the initial structure are adjusted until convergence is achieved.

[0478] In some embodiments, the first scoring module 1002 is specifically used to:

[0479] According to each prediction sub-score and the current weight of each quality evaluation dimension represented by each branch network, the predicted total score of the current image to be screened is calculated according to the second formula;

[0480] Wherein, the second formula is:

[0481] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0482] f(z) represents the predicted total score of the current image to be screened; k represents the number of quality evaluation dimensions; f(x1) represents the predicted sub-score of the current image to be screened on the first quality evaluation dimension, and α represents the current weight of the first quality evaluation dimension; f(x i ) represents the predicted sub-score of the current image to be screened on the i-th quality evaluation dimension, β represents the current weight of the i-th quality evaluation dimension; f(x k ) represents the predicted sub-score of the current image to be screened on the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0483] In some embodiments, the quality evaluation dimensions include at least one of the following: image clarity, image visibility, a pitch angle of a specified object in the image, a heading angle of a specified object in the image, and an area ratio of a specified object in the image.

[0484] Based on the same inventive concept, the present application embodiment provides an image recognition device. Fig.11 , Fig.11 A structural diagram of an image recognition device provided in an embodiment of the present application, the device comprising:

[0485] The to-be-recognized image acquisition module 1101 is used to acquire the to-be-recognized image;

[0486] A first feature extraction module 1102 is used to extract features of the image to be identified to obtain features of the image to be identified;

[0487] The recognition result determination module 1103 is used to obtain the recognition result of the image to be recognized based on the matching result between the image features to be recognized and the image features of the images in the current image library; wherein the images in the current image library are obtained based on the image screening method described in the first aspect above.

[0488] In some embodiments, the apparatus further comprises:

[0489] A first similarity calculation module is used to calculate the fifth similarities between the image feature to be identified and the cluster centers of the current third feature clusters before obtaining the recognition result of the image to be identified based on the matching result between the image feature to be identified and the image features of the images in the current image library; wherein the current third feature clusters are obtained by clustering the image features of the images included in the current third image set; and the images in the current third image set do not include the specified object;

[0490] A first feature cluster selection module is used to select a third feature cluster with the largest fifth similarity from the current third feature clusters;

[0491] A second similarity calculation module, used for calculating each sixth similarity between the image feature to be identified and each image feature included in the currently selected third feature cluster;

[0492] A second object determination module, configured to determine whether the image to be identified contains a specified object according to each sixth similarity corresponding to the image to be identified;

[0493] The recognition result determination module 1103 is specifically used for:

[0494] In the case that the image to be identified contains a designated object, a recognition result of the image to be identified is obtained based on a matching result between features of the image to be identified and image features of images in a current image library.

[0495] In some embodiments, the second object determination module is specifically configured to:

[0496] Determine a third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified;

[0497] If the third preset number of largest sixth similarities are all greater than the third preset similarity threshold, it is determined that the image to be identified does not contain the designated object; otherwise, it is determined that the image to be identified contains the designated object.

[0498] In some embodiments, the apparatus further comprises:

[0499] The third image set updating module is used to add the image to be identified to the current third image set when the image to be identified does not contain the specified object.

[0500] In the technical solution of the present application, operations such as acquisition, storage, use, processing, transmission, provision and disclosure of user facial information are all carried out with the user's authorization.

[0501] It should be noted that the facial images in this embodiment come from a public dataset.

[0502] The present application also provides an electronic device, such as Fig.12 As shown, including:

[0503] Memory 1201, used for storing computer programs;

[0504] The processor 1202 is used to implement any of the steps of the above-mentioned image screening method or image recognition method when executing the program stored in the memory 1201.

[0505] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 1202, the communication interface, and the memory 1201 communicate with each other via the communication bus.

[0506] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0507] The communication interface is used for communication between the above electronic device and other devices.

[0508] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0509] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0510] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image screening methods or image recognition methods are implemented.

[0511] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image screening method or image recognition method in the above-mentioned embodiments.

[0512] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0513] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0514] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0515] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image screening method, characterized in that: The method comprises: Select an image that is not currently being screened from all the images to be screened as the current image to be screened; The current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and the predicted total score of the current image to be screened is calculated according to each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein the quality assessment model is obtained by training based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality assessment dimension; If the predicted total score of the current image to be screened belongs to the preset range, the current weights of each quality evaluation dimension are adjusted based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio, and when the predicted total score of the current image to be screened is not less than the preset filtering threshold, the screening result of the current image to be screened is determined to be retained, and when the predicted total score of the current image to be screened is less than the preset filtering threshold, the screening result of the current image to be screened is determined to be not retained; wherein, the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose screening results are not retained to the number of images whose screening results have been determined at present; Returning to the step of selecting an image that is not currently being screened from all the images to be screened as the current image to be screened, until the screening results of all the images to be screened are obtained; Based on the images that are retained as a result of the screening, images for adding to the current image library are obtained.

2. The method according to claim 1, characterized in that The step of obtaining an image for adding to the current image library based on the image that is retained according to the screening result includes: Select an image that is not currently processed from the images that are retained in the screening results as the current image to be processed; Extract features of the current image to be processed to obtain the current first image features; Based on the matching results of the current first feature clusters and the current first image features, determining whether the current image to be processed contains the specified object; wherein the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; and the images in the current first image set do not contain the specified object; If the current image to be processed contains the specified object, the current image to be processed is determined to be an image to be added to the current image library, and the step of returning to execute the step of selecting a currently unprocessed image from the images retained as a result of the screening as the current image to be processed.

3. The method according to claim 2, characterized in that The determining whether the current image to be processed contains the specified object based on the matching results of the current first feature clusters and the current first image features includes: Calculating first similarities between the current first image feature and the cluster centers of the current first feature clusters; From the current first feature clusters, select the first feature cluster with the largest first similarity; Calculating each second similarity between the current first image feature and each image feature included in the currently selected first feature cluster; According to each second similarity corresponding to the current image to be processed, it is determined whether the current image to be processed contains the designated object.

4. The method according to claim 3, characterized in that The step of determining whether the current image to be processed includes the specified object according to each second similarity corresponding to the current image to be processed includes: Determine a first preset number of largest second similarities among the second similarities corresponding to the current image to be processed; if the first preset number of largest second similarities are all greater than a first preset similarity threshold, determine that the current image to be processed does not contain the specified object; otherwise, determine that the current image to be processed contains the specified object; and / or, The method further includes: when it is determined that the current image to be processed does not contain the designated object, adding the current image to be processed to the current first image set.

5. The method according to claim 1, characterized in that Before inputting the current image to be screened into the pre-trained quality assessment model to obtain the prediction sub-score output by each branch network in the quality assessment model, the method further includes: Perform feature extraction on the current image to be screened to obtain the current second image feature; Based on the matching results of the current second feature clusters and the current second image features, determining whether the current image to be screened contains the specified object; wherein the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; and the images in the current second image set do not contain the specified object; The step of inputting the current image to be screened into a pre-trained quality assessment model to obtain the prediction sub-score output by each branch network in the quality assessment model comprises: In the case that the current image to be screened contains the designated object, the current image to be screened is input into a pre-trained quality assessment model to obtain a prediction sub-score output by each branch network in the quality assessment model.

6. The method according to claim 5, characterized in that The determining whether the current image to be screened contains the specified object based on the matching results of the current second feature clusters and the current second image features includes: Calculating third similarities between the current second image feature and the cluster centers of the current second feature clusters; From the current second feature clusters, select the second feature cluster with the largest third similarity; Calculating each fourth similarity between the current second image feature and each image feature included in the currently selected second feature cluster; It is determined whether the current image to be screened contains the designated object according to each fourth similarity corresponding to the current image to be screened.

7. The method according to claim 6, characterized in that The step of determining whether the current image to be screened contains the specified object according to each fourth similarity corresponding to the current image to be screened includes: Determine a second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened; if the second preset number of largest fourth similarities are all greater than the second preset similarity threshold, determine that the current image to be screened does not contain the specified object; otherwise, determine that the current image to be screened contains the specified object; and / or, The method further comprises: When it is determined that the current image to be filtered does not contain the specified object, the current image to be filtered is added to the current second image set, and the step of selecting an image that is not currently filtered from all the images to be filtered is returned to be executed as the current image to be filtered.

8. The method according to claim 1, characterized in that The current weights of the quality evaluation dimensions are adjusted based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio, including: Get the current status; the current status includes: the current filtering ratio, the number of images for which the filtering results have been determined, and the current weight of each quality evaluation dimension; Determine the expected benefit when taking each alternative action in the current state as the current first expectation; wherein an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension; Select the alternative action corresponding to the current largest first expectation as the action to be taken currently; Based on the current actions that need to be taken, adjust the current weights of each quality evaluation dimension; The method further comprises: After determining the action currently required to be taken, updating the current first expectation based on the first formula; Among them, the first formula is: Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A)); Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken at present; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after the screening result of the current image to be screened is obtained, maxQ(S′,A′) represents the maximum value of the second expectation of the benefit obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value in the second expectation; st represents the constraint condition, R(S,A) represents: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c represents the preset filtering threshold, R min Indicates the minimum value in the preset range, R max represents the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other situations.

9. The method according to claim 1, characterized in that: The quality evaluation model is trained by the following steps: Acquire a sample image containing a specified object; wherein the label of the sample image comprises: a sample sub-score of the sample image in each quality evaluation dimension, and a sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and weights of each quality evaluation dimension of the sample image; for each sample image, input the sample image into a quality evaluation model of an initial structure, obtain the training sub-scores output by each branch network in the quality evaluation model of the initial structure, and calculate the training total score of the sample image based on the initial values ​​of the weights of each quality evaluation dimension represented by each training sub-score and each branch network; calculate a loss value based on each training sub-score and the training total score of the sample image, as well as each sample sub-score and the sample total score of the sample image; adjust the model parameters of the quality evaluation model of the initial structure based on the obtained loss value until convergence is reached; and / or, The step of calculating the predicted total score of the current image to be screened according to the current weights of the quality evaluation dimensions represented by the prediction sub-scores and the branch networks includes: calculating the predicted total score of the current image to be screened according to the second formula according to the current weights of the quality evaluation dimensions represented by the prediction sub-scores and the branch networks; wherein the second formula is: f ( z ) = w x ( α f ( x 1 ) x ... x β f ( x i ) × … × εf(x k ))+n: f(z) represents the predicted total score of the current image to be screened; k represents the number of quality evaluation dimensions; f(x1) represents the predicted sub-score of the current image to be screened on the first quality evaluation dimension, and α represents the current weight of the first quality evaluation dimension; f(x i ) represents the predicted sub-score of the current image to be screened on the i-th quality evaluation dimension, β represents the current weight of the i-th quality evaluation dimension; f(x k ) represents the predicted sub-score of the current image to be screened in the kth quality evaluation dimension, ε represents the current weight of the kth quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter; and / or, The quality evaluation dimensions include at least one of the following: image clarity, image visibility, a pitch angle of a specified object in the image, a heading angle of a specified object in the image, and an area ratio of a specified object in the image.

10. An image recognition method, characterized in that: The method comprises: Obtain an image to be recognized; Extracting features of the image to be identified to obtain features of the image to be identified; Based on the matching result between the features of the image to be identified and the image features of the images in the current image library, the recognition result of the image to be identified is obtained; wherein the images in the current image library are obtained based on the image screening method described in any one of claims 1-9.

11. The method according to claim 10, characterized in that Before obtaining the recognition result of the image to be recognized based on the matching result between the image features to be recognized and the image features of the images in the current image library, the method further includes: Calculating the fifth similarities between the features of the image to be identified and the cluster centers of the current third feature clusters; wherein the current third feature clusters are obtained by clustering the image features of the images included in the current third image set; the images in the current third image set do not contain the specified object; selecting the third feature cluster with the largest corresponding fifth similarity from the current third feature clusters; calculating the sixth similarities between the features of the image to be identified and the image features included in the currently selected third feature cluster; determining whether the image to be identified contains the specified object based on the sixth similarities corresponding to the image to be identified; obtaining the recognition result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library, including: obtaining the recognition result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library when the image to be identified contains the specified object; and / or, The determining, based on the sixth similarities corresponding to the image to be identified, whether the image to be identified contains the designated object comprises: determining a third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified; if the third preset number of largest sixth similarities are all greater than a third preset similarity threshold, determining that the image to be identified does not contain the designated object; otherwise, determining that the image to be identified contains the designated object; and / or, The method further includes: if the image to be identified does not contain a designated object, adding the image to be identified to a current third image set.

12. An image screening device, characterized in that: The device comprises: A first image selection module is used to select an image that is not currently being screened from all images to be screened as the current image to be screened; The first scoring module is used to input the current image to be screened into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the current image to be screened according to each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein the quality assessment model is obtained by training based on the sample sub-scores of the sample images containing the specified objects on each quality assessment dimension and the sample total score of the sample images; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores of the sample image and the weights of each quality assessment dimension; A screening result determination module is used to adjust the current weights of each quality evaluation dimension based on the reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio if the predicted total score of the current image to be screened belongs to a preset range, and to determine that the screening result of the current image to be screened is retained when the predicted total score of the current image to be screened is not less than the preset filtering threshold, and to determine that the screening result of the current image to be screened is not retained when the predicted total score of the current image to be screened is less than the preset filtering threshold; wherein the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose screening results are not retained to the number of images whose screening results have been determined at present; and return to trigger the first image selection module until the screening results of all images to be screened are obtained; The image adding module is used to obtain images to be added to the current image library based on the images that are retained as a result of the screening.

13. An image recognition device, characterized in that: The device comprises: An image acquisition module to be identified, used to acquire an image to be identified; A first feature extraction module is used to extract features of the image to be identified to obtain features of the image to be identified; A recognition result determination module is used to obtain a recognition result of the image to be recognized based on a matching result between the features of the image to be recognized and the image features of the images in the current image library; wherein the images in the current image library are obtained based on the image screening method described in any one of claims 1-9.

14. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-9, or any of the methods described in claims 10-11, when executing a program stored in a memory.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 9, or any one of claims 10 to 11 is implemented.

Images