An image screening method, an image recognition method, an apparatus, and an electronic device

Abstract

Embodiments of the present application provide an image screening method, an image recognition method, an image screening device, an image recognition device and an electronic device, and relate to the technical field of computer vision. The method comprises: selecting an image that has not been screened from all images to be screened as a current image to be screened; inputting the current image to be screened into a pre-trained quality evaluation model to obtain predicted sub-scores, and calculating a predicted total score of the current image to be screened according to the predicted sub-scores and current weights of each quality evaluation dimension represented by each branch network; if the predicted total score of the current image to be screened belongs to a preset range, adjusting the current weights of each quality evaluation dimension based on a reinforcement learning algorithm according to a current filtering ratio and a target filtering ratio, and determining that a screening result of the current image to be screened is to be retained in a case where the predicted total score of the current image to be screened is not less than a preset filtering threshold. In this way, the efficiency of image screening can be improved.

Description

An image screening method, image recognition method, apparatus, and electronic device. Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image screening method, an image recognition method, an apparatus, and an electronic device. Background Technology

[0002] With the development of computer vision technology, the technique of detecting objects in images through image feature matching has been widely applied. Correspondingly, electronic devices typically pre-store a subset of labeled images. For example, an electronic device can store labeled images using a pre-set image library. The label of any image indicates the identity of the object within that image. Furthermore, in practical applications, when an electronic device receives an image to be identified, it can compare the image features of the image to be identified with the image features of the pre-stored images to determine the identity of the object in the image to be identified. If the quality of the pre-stored images is low, the identification result of the image to be identified may be inaccurate. One approach is to manually screen the images to select high-quality images for storage.

[0003] However, manual screening is not very efficient. Summary of the Invention

[0004] The purpose of this application is to provide an image filtering method, image recognition method, apparatus, and electronic device to improve the efficiency of image filtering. The specific technical solution is as follows:

[0005] A first aspect of the embodiments of this application provides an image filtering method, the method comprising:

[0006] Select one image from all images to be filtered that is not currently being filtered, and use it as the current image to be filtered.

[0007] The image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model. Based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network, the total predicted score of the image to be screened is calculated. The quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of that sample image. The total sample score of a sample image is obtained based on the initial values ​​of each sample sub-score and the weights of each quality assessment dimension of that sample image.

[0008] If the predicted total score of the current image to be filtered falls within a preset range, then based on the current filtering ratio and the target filtering ratio, the current weights of each quality evaluation dimension are adjusted using a reinforcement learning algorithm. If the predicted total score of the current image to be filtered is not less than a preset filtering threshold, the image to be filtered is determined to be retained; if the predicted total score of the current image to be filtered is less than the preset filtering threshold, the image to be filtered is determined to be not retained. The preset range includes the preset filtering threshold. The current filtering ratio is the ratio of the number of images whose filtering result is not retained to the number of images whose filtering result has been determined.

[0009] Return to the step of selecting an image that has not yet been filtered from all images to be filtered as the current image to be filtered, until the filtering results of all images to be filtered are obtained;

[0010] Based on the images selected for retention, images are obtained for addition to the current image library.

[0011] In some embodiments, obtaining images for addition to the current image library based on the images selected for retention after filtering includes:

[0012] Select one currently unprocessed image from the images that are selected as the current image to be processed;

[0013] Extract features from the current image to be processed to obtain the current first image features;

[0014] Based on the matching results of the current first feature clusters and the current first image features, it is determined whether the current image to be processed contains the specified object; wherein, the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; the images in the current first image set do not contain the specified object;

[0015] If the specified object is contained in the current image to be processed, then the current image to be processed is determined to be the image to be added to the current image library, and the process returns to the step of selecting an unprocessed image from the images that are selected as the current image to be processed.

[0016] In some embodiments, determining whether the specified object is contained in the current image to be processed based on the matching results of the current first feature clusters and the current first image features includes:

[0017] Calculate the first similarity between the current first image feature and the cluster centers of each current first feature cluster;

[0018] From the current first feature clusters, select the first feature cluster with the highest first similarity;

[0019] Calculate the second similarity between the current first image feature and each image feature contained in the currently selected first feature cluster;

[0020] Based on the second similarity scores corresponding to the current image to be processed, determine whether the current image to be processed contains the specified object.

[0021] In some embodiments, determining whether the current image to be processed contains a specified object based on each second similarity corresponding to the current image to be processed includes:

[0022] Determine the largest number of second similarities among the various second similarities corresponding to the current image to be processed;

[0023] If all of the first preset number of largest second similarities are greater than the first preset similarity threshold, then it is determined that the current image to be processed does not contain the specified object; otherwise, it is determined that the current image to be processed contains the specified object.

[0024] In some embodiments, the method further includes:

[0025] If it is determined that the specified object is not contained in the current image to be processed, the current image to be processed is added to the current first image set.

[0026] In some embodiments, before inputting the currently selected image into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, the method further includes:

[0027] Feature extraction is performed on the current image to be filtered to obtain the current second image features;

[0028] Based on the matching results of the current second feature clusters and the current second image features, it is determined whether the current image to be filtered contains the specified object; wherein, the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; the images in the current second image set do not contain the specified object;

[0029] The step of inputting the image to be screened into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model includes:

[0030] If the specified object is contained in the current image to be screened, the current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model.

[0031] In some embodiments, determining whether the specified object is contained in the current image to be filtered based on the matching results of the current second feature clusters and the current second image features includes:

[0032] Calculate the third similarity between the current second image feature and the cluster centers of each of the current second feature clusters;

[0033] From the current second feature clusters, select the second feature cluster with the highest corresponding third similarity;

[0034] Calculate the fourth similarity between the current second image feature and each image feature contained in the currently selected second feature cluster;

[0035] Based on the fourth similarity scores of the current image to be filtered, determine whether the current image to be filtered contains the specified object.

[0036] In some embodiments, determining whether the current image to be filtered contains a specified object based on each fourth similarity corresponding to the current image to be filtered includes:

[0037] Determine the largest fourth similarity scores among the fourth similarity scores corresponding to the current image to be screened, which is a second preset number.

[0038] If all of the second preset number of largest fourth similarities are greater than the second preset similarity threshold, then it is determined that the current image to be filtered does not contain the specified object; otherwise, it is determined that the current image to be filtered contains the specified object.

[0039] In some embodiments, the method further includes:

[0040] If it is determined that the specified object is not contained in the current image to be filtered, the current image to be filtered is added to the current second image set, and the process returns to the step of selecting an image that has not yet been filtered from all the images to be filtered as the current image to be filtered.

[0041] In some embodiments, adjusting the current weights of each quality evaluation dimension based on a reinforcement learning algorithm according to the current filtering ratio and the target filtering ratio includes:

[0042] Obtain the current state; where the current state includes: the current filtering ratio, the number of images that have been determined as the filtering results, and the current weight of each quality evaluation dimension;

[0043] Determine the expected benefit of taking each alternative action in the current state, as the current first expectation; wherein, an alternative action includes: the adjustment amount of the weights of each quality evaluation dimension;

[0044] Select the alternative action corresponding to the highest current first expectation as the action to be taken now;

[0045] Based on the actions required at present, the current weights of each quality evaluation dimension will be adjusted.

[0046] In some embodiments, the method further includes:

[0047] After determining the action that needs to be taken, update the current first expectation based on the first formula;

[0048] The first formula is:

[0049] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0050]

[0051] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after obtaining the filtering result of the current image to be filtered, maxQ(S′,A′) represents the maximum value of the second expectation of the reward obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value of the second expectation;

[0052] st represents the constraint condition, R(S,A) means: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c R represents the preset filtering threshold. min R represents the minimum value within the preset range. max R represents the maximum value within the preset range. min <R c <R max M represents the target filtering ratio, p represents the current filtering ratio, and otherwise represents other cases.

[0053] In some embodiments, the quality assessment model is trained through the following steps:

[0054] Obtain a sample image containing a specified object; wherein the label of the sample image includes: the sample sub-score of the sample image on each quality evaluation dimension, and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and the weights of each quality evaluation dimension of the sample image.

[0055] For each sample image, the sample image is input into the quality assessment model of the initial structure to obtain the training sub-scores output by each branch network in the quality assessment model of the initial structure. Based on the initial values ​​of the weights of each quality assessment dimension represented by each branch network, the total training score of the sample image is calculated.

[0056] The loss value is calculated based on the training sub-scores and total training score of the sample image, as well as the sample sub-scores and total sample score of the sample image.

[0057] The model parameters of the initial structure quality evaluation model are adjusted based on the obtained loss value until convergence is achieved.

[0058] In some embodiments, calculating the total predicted score of the image to be screened based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network includes:

[0059] Based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network, the total predicted score of the image to be screened is calculated according to the second formula.

[0060] The second formula is:

[0061] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0062] f(z) represents the predicted total score of the image to be selected; k represents the number of each quality assessment dimension; f(x1) represents the predicted sub-score of the image to be selected in the first quality assessment dimension, and α represents the current weight of the first quality assessment dimension; f(x i f(x) represents the predicted sub-rating of the image to be filtered in the i-th quality assessment dimension, and β represents the current weight of the i-th quality assessment dimension; k ) represents the predicted sub-score of the image to be filtered in the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0063] In some embodiments, each quality evaluation dimension includes at least one of the following: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image.

[0064] A second aspect of this application provides an image recognition method, the method comprising:

[0065] Acquire the image to be recognized;

[0066] Feature extraction is performed on the image to be identified to obtain the image features;

[0067] Based on the matching result between the features of the image to be identified and the image features of the images in the current image library, the identification result of the image to be identified is obtained; wherein, the images in the current image library are obtained based on any of the image filtering methods described in the first aspect above.

[0068] In some embodiments, before obtaining the recognition result of the image to be recognized based on the matching result between the features of the image to be recognized and the image features of images in the current image library, the method further includes:

[0069] Calculate the fifth similarity between the features of the image to be identified and the cluster centers of each of the current third feature clusters; wherein each of the current third feature clusters is obtained by clustering the image features of the images contained in the current third image set; the images in the current third image set do not contain the specified object;

[0070] From the current third feature clusters, select the third feature cluster with the highest corresponding fifth similarity;

[0071] Calculate the sixth similarity between the features of the image to be identified and each image feature contained in the currently selected third feature cluster;

[0072] Based on the sixth similarity scores corresponding to the image to be identified, determine whether the image to be identified contains the specified object;

[0073] The process of obtaining the recognition result of the image to be recognized based on the matching result between the features of the image to be recognized and the image features of images in the current image database includes:

[0074] If the image to be identified contains a specified object, the identification result of the image to be identified is obtained based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0075] In some embodiments, determining whether the image to be identified contains a specified object based on each sixth similarity corresponding to the image to be identified includes:

[0076] Determine the largest sixth similarity among the sixth similarities corresponding to the image to be identified, which is a third preset number.

[0077] If all of the third preset number of largest sixth similarities are greater than the third preset similarity threshold, then it is determined that the image to be identified does not contain the specified object; otherwise, it is determined that the image to be identified contains the specified object.

[0078] In some embodiments, the method further includes:

[0079] If the image to be identified does not contain the specified object, the image to be identified is added to the current third image set.

[0080] A third aspect of this application provides an image filtering device, the device comprising:

[0081] The first image selection module is used to select one image that has not yet been filtered from all images to be filtered, and use it as the current image to be filtered.

[0082] The first scoring module is used to input the image to be screened into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the image to be screened based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network. The quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of that sample image. The total sample score of a sample image is obtained based on the initial values ​​of each sample sub-score and the weights of each quality assessment dimension of that sample image.

[0083] The filtering result determination module is used to determine the filtering result of the current image to be filtered as "retained" if the predicted total score of the current image to be filtered falls within a preset range, and to adjust the current weights of each quality evaluation dimension based on the current filtering ratio and the target filtering ratio using a reinforcement learning algorithm. If the predicted total score of the current image to be filtered is not less than a preset filtering threshold, the module determines that the image to be filtered is "not retained"; if the predicted total score of the current image to be filtered is less than the preset filtering threshold, the module determines that the image to be filtered is "not retained". The preset range includes the preset filtering threshold. The current filtering ratio is the ratio of the number of images whose filtering result is "not retained" to the number of images whose filtering result has been determined. The module then returns to trigger the first image selection module until the filtering results of all images to be filtered are obtained.

[0084] The image adding module is used to obtain images to be added to the current image library based on the images selected for retention from the filtering results.

[0085] In some embodiments, the image adding module includes:

[0086] The first image selection submodule is used to select one currently unprocessed image from the images that are selected as the current image to be processed;

[0087] The first feature extraction submodule is used to extract features from the current image to be processed, and obtain the current first image features;

[0088] The first object determination submodule is used to determine whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features; wherein, the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; the images in the current first image set do not contain the specified object;

[0089] The first image addition submodule is used to determine that if the image to be processed contains the specified object, the image to be processed is an image to be added to the current image library, and to trigger the first image selection submodule.

[0090] In some embodiments, the first object determining submodule includes:

[0091] The first similarity calculation unit is used to calculate the first similarity between the current first image feature and the cluster centers of the current first feature clusters;

[0092] The first feature cluster selection unit is used to select the first feature cluster with the highest first similarity from the current first feature clusters;

[0093] The second similarity calculation unit is used to calculate the second similarity between the current first image feature and each image feature contained in the currently selected first feature cluster;

[0094] The first object determination unit is used to determine whether the current image to be processed contains a specified object based on the second similarity scores corresponding to the current image to be processed.

[0095] In some embodiments, the first object determining unit is specifically used for:

[0096] Determine the largest number of second similarities among the various second similarities corresponding to the current image to be processed;

[0097] If all of the first preset number of largest second similarities are greater than the first preset similarity threshold, then it is determined that the current image to be processed does not contain the specified object; otherwise, it is determined that the current image to be processed contains the specified object.

[0098] In some embodiments, the apparatus further includes:

[0099] The first image set update module is used to add the currently pending image to the current first image set if it is determined that the specified object is not contained in the currently pending image.

[0100] In some embodiments, the apparatus further includes:

[0101] The second feature extraction module is used to extract features from the current image to be screened before inputting the current image to be screened into the pre-trained quality assessment model and obtaining the predicted sub-scores output by each branch network in the quality assessment model, so as to obtain the current second image features.

[0102] The first object determination module is used to determine whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features; wherein, the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; the images in the current second image set do not contain the specified object;

[0103] The first scoring module is specifically used for:

[0104] If the specified object is contained in the current image to be screened, the current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model.

[0105] In some embodiments, the first object determining module includes:

[0106] The first similarity calculation submodule is used to calculate the third similarity between the current second image feature and the cluster centers of each of the current second feature clusters;

[0107] The first feature cluster selection submodule is used to select the corresponding second feature cluster with the highest third similarity from the current second feature clusters;

[0108] The second similarity calculation submodule is used to calculate the fourth similarity between the current second image feature and each image feature contained in the currently selected second feature cluster;

[0109] The second object determination submodule is used to determine whether the image to be filtered contains a specified object based on the fourth similarity scores corresponding to the image to be filtered.

[0110] In some embodiments, the second object determining submodule is specifically used for:

[0111] Determine the largest fourth similarity scores among the fourth similarity scores corresponding to the current image to be screened, which is a second preset number.

[0112] If all of the second preset number of largest fourth similarities are greater than the second preset similarity threshold, then it is determined that the current image to be filtered does not contain the specified object; otherwise, it is determined that the current image to be filtered contains the specified object.

[0113] In some embodiments, the apparatus further includes:

[0114] The second image set update module is used to add the currently selected image to the current second image set and trigger the first image selection module when it is determined that the currently selected image does not contain the specified object.

[0115] In some embodiments, the filtering result determination module is specifically used for:

[0116] Obtain the current state; where the current state includes: the current filtering ratio, the number of images that have been determined as the filtering results, and the current weight of each quality evaluation dimension;

[0117] Determine the expected benefit of taking each alternative action in the current state, as the current first expectation; wherein, an alternative action includes: the adjustment amount of the weights of each quality evaluation dimension;

[0118] Select the alternative action corresponding to the highest current first expectation as the action to be taken now;

[0119] Based on the actions required at present, the current weights of each quality evaluation dimension will be adjusted.

[0120] In some embodiments, the apparatus further includes:

[0121] The expectation update module is used to update the current first expectation based on the first formula after determining the action that needs to be taken.

[0122] The first formula is:

[0123] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0124]

[0125] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after obtaining the filtering result of the current image to be filtered, maxQ(S′,A′) represents the maximum value of the second expectation of the reward obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value of the second expectation;

[0126] st represents the constraint condition, R(S,A) means: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c R represents the preset filtering threshold. min R represents the minimum value within the preset range. max R represents the maximum value within the preset range. min <R c <R max M represents the target filtering ratio, p represents the current filtering ratio, and otherwise represents other cases.

[0127] In some embodiments, the quality assessment model is trained through the following steps:

[0128] Obtain a sample image containing a specified object; wherein the label of the sample image includes: the sample sub-score of the sample image on each quality evaluation dimension, and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and the weights of each quality evaluation dimension of the sample image.

[0129] For each sample image, the sample image is input into the quality assessment model of the initial structure to obtain the training sub-scores output by each branch network in the quality assessment model of the initial structure. Based on the initial values ​​of the weights of each quality assessment dimension represented by each branch network, the total training score of the sample image is calculated.

[0130] The loss value is calculated based on the training sub-scores and total training score of the sample image, as well as the sample sub-scores and total sample score of the sample image.

[0131] The model parameters of the initial structure quality evaluation model are adjusted based on the obtained loss value until convergence is achieved.

[0132] In some embodiments, the first scoring module is specifically used for:

[0133] Based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network, the total predicted score of the image to be screened is calculated according to the second formula.

[0134] The second formula is:

[0135] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0136] f(z) represents the predicted total score of the image to be selected; k represents the number of each quality assessment dimension; f(x1) represents the predicted sub-score of the image to be selected in the first quality assessment dimension, and α represents the current weight of the first quality assessment dimension; f(x i f(x) represents the predicted sub-rating of the image to be filtered in the i-th quality assessment dimension, and β represents the current weight of the i-th quality assessment dimension; k ) represents the predicted sub-score of the image to be filtered in the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0137] In some embodiments, each quality evaluation dimension includes at least one of the following: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image.

[0138] A fourth aspect of this application provides an image recognition device, the device comprising:

[0139] The image acquisition module is used to acquire the image to be recognized.

[0140] The first feature extraction module is used to extract features from the image to be identified, thereby obtaining the features of the image to be identified;

[0141] The recognition result determination module is used to obtain the recognition result of the image to be recognized based on the matching result between the features of the image to be recognized and the image features of the images in the current image library; wherein the images in the current image library are obtained based on the image filtering method described in the first aspect above.

[0142] In some embodiments, the apparatus further includes:

[0143] The first similarity calculation module is used to calculate the fifth similarity between the features of the image to be identified and the cluster centers of each of the current third feature clusters before obtaining the recognition result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library; wherein, each of the current third feature clusters is obtained by clustering the image features of the images contained in the current third image set; the images in the current third image set do not contain the specified object;

[0144] The first feature cluster selection module is used to select the third feature cluster with the highest similarity from the current third feature clusters;

[0145] The second similarity calculation module is used to calculate the sixth similarity between the features of the image to be identified and the image features contained in the currently selected third feature cluster;

[0146] The second object determination module is used to determine whether the image to be identified contains a specified object based on each sixth similarity corresponding to the image to be identified.

[0147] The identification result determination module is specifically used for:

[0148] If the image to be identified contains a specified object, the identification result of the image to be identified is obtained based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0149] In some embodiments, the second object determining module is specifically used for:

[0150] Determine the largest sixth similarity among the sixth similarities corresponding to the image to be identified, which is a third preset number.

[0151] If all of the third preset number of largest sixth similarities are greater than the third preset similarity threshold, then it is determined that the image to be identified does not contain the specified object; otherwise, it is determined that the image to be identified contains the specified object.

[0152] In some embodiments, the apparatus further includes:

[0153] The third image set update module is used to add the image to be identified to the current third image set if the image to be identified does not contain the specified object.

[0154] A fifth aspect of this application provides an electronic device, including:

[0155] Memory, used to store computer programs;

[0156] When a processor executes a program stored in memory, it implements any of the image filtering methods or image recognition methods described above.

[0157] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the image filtering methods or image recognition methods described above.

[0158] In another aspect of the embodiments of this application, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform any of the image filtering methods or image recognition methods described above.

[0159] Beneficial effects of the embodiments in this application:

[0160] Based on the image filtering method provided in this application, an image that has not yet been filtered can be selected from all images to be filtered as the current image to be filtered, and this current image to be filtered is input into a pre-trained quality assessment model. The quality assessment model contains multiple branch networks, each branch network representing a quality assessment dimension. Accordingly, the quality assessment model can output the sub-scores (i.e., predicted sub-scores) of the current image to be filtered on each quality assessment dimension. Then, the predicted total score of the current image to be filtered can be calculated based on each predicted sub-score and the current weight of each quality assessment dimension. Based on the predicted total score of the current image to be filtered, the filtering result (retain or not retain) of the current image to be filtered is determined. Accordingly, the filtering result of each image among all images to be filtered can be obtained by following the above steps. In this way, automatic scoring and filtering of each image to be filtered can be achieved through a pre-trained quality assessment model, eliminating the need for manual filtering of images to be filtered and improving filtering efficiency.

[0161] After obtaining the filtering results for each image to be filtered, since the filtering results are determined based on the image's predicted total score, which reflects the overall quality of the image across multiple quality evaluation dimensions, the images selected for inclusion in the current image library based on the filtering results ensure that only high-quality images are added, facilitating subsequent processing.

[0162] Furthermore, during the image filtering process, the target filtering ratio and the current filtering ratio can be combined, and the weights of each quality evaluation dimension can be dynamically adjusted based on a reinforcement learning algorithm. In other words, at each stage of filtering all images to be filtered, the current filtering ratio can be continuously adjusted towards the target filtering ratio. This allows for filtering of all images according to actual filtering needs, ensuring a balanced ratio between the number of retained and excluded images, thus meeting the user's personalized requirements.

[0163] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0164] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0165] Figure 1 is a flowchart of a first type of image filtering method provided in an embodiment of this application;

[0166] Figure 2 is a second flowchart of the image filtering method provided in an embodiment of this application;

[0167] Figure 3 is a flowchart of an embodiment of this application for determining whether a specified object is contained in the current image to be processed;

[0168] Figure 4 is a third flowchart of the image filtering method provided in the embodiments of this application;

[0169] Figure 5 is a flowchart of adjusting the current weights of each quality evaluation dimension according to an embodiment of this application;

[0170] Figure 6 is a flowchart of a training quality evaluation model provided in an embodiment of this application;

[0171] Figure 7 is a flowchart of a first type of image recognition method provided in an embodiment of this application;

[0172] Figure 8 is a second flowchart of the image recognition method provided in an embodiment of this application;

[0173] Figure 9 is a schematic diagram of an image detection process provided in an embodiment of this application;

[0174] Figure 10 is a structural diagram of an image filtering device provided in an embodiment of this application;

[0175] Figure 11 is a structural diagram of an image recognition device provided in an embodiment of this application;

[0176] Figure 12 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0177] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0178] With the development of computer vision technology, the technique of detecting objects in images through image feature matching has been widely applied. Correspondingly, electronic devices typically pre-store a subset of labeled images. For example, an electronic device can store labeled images using a pre-set image library. The label of any image indicates the identity of the object within that image. Furthermore, in practical applications, when an electronic device receives an image to be identified, it can compare the image features of the image to be identified with the image features of the pre-stored images to determine the identity of the object in the image to be identified. If the quality of the pre-stored images is low, the identification result of the image to be identified may be inaccurate. One approach is to manually screen the images to select high-quality images for storage.

[0179] However, manual screening is not very efficient.

[0180] The image filtering method provided in this application can be applied to scenarios such as identity authentication and category recognition. Accordingly, the type of the specified object can be different in different application scenarios.

[0181] For example, in an identity authentication scenario, the designated object can be a person's face. Accordingly, a user (such as a resident of a community) can upload a facial photo and the identity information of a person authorized to enter a designated area (such as the community) via a smart terminal. The electronic device can then save this photo to a pre-set image library. Subsequently, the electronic device can acquire an image of the face of the person needing to enter the designated area through a turnstile at the entrance, using it as the image to be identified. Then, the electronic device can compare the image features of the acquired image to the image features of each image in the image library to determine whether the person needing to enter the designated area has the necessary permissions.

[0182] For example, in an identity authentication scenario, the designated object can be a human body. Accordingly, users (such as residents of a community) can upload a full-body photo of a person authorized to enter a designated area (such as the community) along with that person's identity information via a smart terminal. The electronic device can then save this photo to a pre-set image library. Subsequently, the electronic device can obtain an image of the person currently attempting to enter the designated area through a turnstile at the entrance, using it as the image to be identified. Furthermore, the electronic device can compare the image features of the acquired image to those of every image in the image library to determine whether the person attempting to enter the designated area has the necessary permissions.

[0183] It is understandable that user-uploaded images may have issues such as blurriness, distortion, or objects appearing too small, meaning the image quality is low. Consequently, this may lead to inaccurate recognition results for the image being identified.

[0184] This application provides an image filtering method that can be applied to electronic devices. Referring to Figure 1, which is a first flowchart of the image filtering method provided in this application, the method includes the following steps:

[0185] S101: Select an image that has not yet been filtered from all images to be filtered, and use it as the current image to be filtered.

[0186] S102: Input the image to be screened into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the total predicted score of the image to be screened based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network.

[0187] The quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of the sample image. The total sample score of a sample image is obtained based on the initial values ​​of the weights of each sample sub-score and each quality assessment dimension of the sample image.

[0188] S103: If the predicted total score of the current image to be screened is within the preset range, then based on the current filtering ratio and the target filtering ratio, the current weights of each quality evaluation dimension are adjusted according to the reinforcement learning algorithm. If the predicted total score of the current image to be screened is not less than the preset filtering threshold, the current image to be screened is determined to be retained. If the predicted total score of the current image to be screened is less than the preset filtering threshold, the current image to be screened is determined to be not retained.

[0189] If the filtering results for all images to be filtered have not been obtained, then return to step S101; if the filtering results for all images to be filtered have been obtained, then proceed to step S104.

[0190] The preset range includes a preset filtering threshold; the current filtering ratio is the ratio of the number of images that are not retained in the filtering result to the number of images that have been determined in the current filtering result.

[0191] S104: Based on the images selected for retention, obtain the images to be added to the current image library.

[0192] Based on the above processing, an unselected image can be chosen from all images to be screened as the current image to be screened, and this image is then input into a pre-trained quality assessment model. The quality assessment model contains multiple branch networks, each representing a quality assessment dimension. Correspondingly, the model outputs sub-scores (i.e., predicted sub-scores) for each quality assessment dimension of the current image to be screened. Then, based on each predicted sub-score and the current weights of each quality assessment dimension, the predicted total score of the current image to be screened is calculated. Based on the predicted total score, the screening result (keeping or not keeping) of the current image to be screened is determined. Similarly, the screening result for each image among all images to be screened can be obtained by following the above steps. In this way, automatic scoring and screening of each image to be screened can be achieved through a pre-trained quality assessment model, eliminating the need for manual screening and improving screening efficiency.

[0193] After obtaining the filtering results for each image to be filtered, since the filtering results are determined based on the image's predicted total score, which reflects the overall quality of the image across multiple quality evaluation dimensions, the images selected for inclusion in the current image library based on the filtering results ensure that only high-quality images are added, facilitating subsequent processing.

[0194] Furthermore, during the image filtering process, the target filtering ratio and the current filtering ratio can be combined, and the weights of each quality evaluation dimension can be dynamically adjusted based on a reinforcement learning algorithm. In other words, at each stage of filtering all images to be filtered, the current filtering ratio can be continuously adjusted towards the target filtering ratio. This allows for filtering of all images according to actual filtering needs, ensuring a balanced ratio between the number of retained and excluded images, thus meeting the user's personalized requirements.

[0195] For step S101, the image to be filtered refers to the image acquired by the electronic device that is to be added to the image library.

[0196] For ease of description, let's take the above-mentioned identity authentication scenario as an example. In the above-mentioned identity authentication scenario, the specified object is a person's face, and the image to be screened can be: a photo of a person with permission to enter the community uploaded by a community resident through a smart terminal.

[0197] In one implementation, during the image filtering process according to steps S101 to S104 described above, the electronic device can acquire new images to be filtered in real time. That is, the total number of images to be filtered is variable during the image filtering process. Thus, during the image filtering process, the electronic device can filter the images acquired in real time, improving the flexibility of image filtering. The specific image filtering process will be described in subsequent embodiments.

[0198] In another implementation, the electronic device can also acquire all images to be added to the image library at once, obtaining all images to be filtered. That is, the total number of images to be filtered is fixed during the image filtering process.

[0199] Furthermore, the electronic device can select one currently unfiltered image from all images to be filtered, and use it as the current image to be filtered.

[0200] In this embodiment, the electronic device can determine the current image to be filtered according to the order in which the images to be filtered are acquired, and process it according to steps S102 to S103. Alternatively, the electronic device can randomly select one image from all the images to be filtered that have not yet been filtered, and process it according to steps S102 to S103. The specific method of determining the current image to be filtered is not limited here.

[0201] Regarding step S102, after determining the image to be filtered, the electronic device can input the image into a pre-trained quality assessment model. This quality assessment model contains multiple branch networks, with each branch network representing a quality assessment dimension. The specific training process of the quality assessment model will be described in subsequent embodiments.

[0202] The quality assessment model can process the input image to be screened, and then each branch network in the quality assessment model can output the predicted sub-score of the image to be screened on its respective quality assessment dimension.

[0203] In one implementation, each quality evaluation dimension includes at least one of the following: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image.

[0204] Image sharpness refers to the clarity and sharpness of the details, textures, and colors in an image; that is, the degree to which the content of the image is recognizable.

[0205] Correspondingly, if an image has high sharpness, it will have a high sub-score on that quality assessment dimension (image sharpness); if an image has low sharpness, it will have a low sub-score on that quality assessment dimension (image sharpness).

[0206] Image visibility represents the degree of occlusion of a specified object in an image. For example, when the specified object is a person's face, the degree of occlusion of the specified object in the image is the proportion of the person's face that is obscured. In other words, the greater the proportion of the person's face that is obscured in the image, the higher the degree of occlusion of the specified object in the image, and the lower the image visibility.

[0207] Correspondingly, if an image has high visibility, it will have a high sub-score on that quality assessment dimension (image visibility); if an image has low visibility, it will have a low sub-score on that quality assessment dimension (image visibility).

[0208] The pitch angle of a specified object in an image represents the distance in the height direction of the lens of the image acquisition device (such as a camera) relative to the specified object. It can be understood that when the lens of the image acquisition device is at the same height as the specified object, the pitch angle of the specified object in the image is 0; the greater the distance in the height direction between the lens of the image acquisition device and the specified object, the larger the pitch angle of the specified object in the image. The smaller the pitch angle of the specified object in the image, the less distortion caused by lens distortion; conversely, the larger the pitch angle of the specified object in the image, the greater the distortion caused by lens distortion.

[0209] Correspondingly, if the pitch angle of a specified object in an image is small, the image will have a high sub-score on that quality assessment dimension (the pitch angle of the specified object in the image); if the pitch angle of a specified object in an image is large, the image will have a low sub-score on that quality assessment dimension (the pitch angle of the specified object in the image).

[0210] The heading angle of a specified object in an image represents the angle between the optical axis of the lens of the image acquisition device (such as a camera) and a specified plane of the specified object in the horizontal direction. For example, when the specified object is a human face, the specified plane can be the plane approximately where the human face is located. In other words, the closer the image acquisition device is to the front of the specified object in the horizontal direction, the smaller the heading angle of the specified object in the image. The smaller the heading angle of the specified object in the image, the more complete the facial information of the captured person; the larger the heading angle of the specified object in the image, the less complete the facial information of the captured person.

[0211] Correspondingly, if the heading angle of a specified object in an image is small, the image will have a high sub-score on that quality assessment dimension (the heading angle of the specified object in the image); if the heading angle of a specified object in an image is large, the image will have a low sub-score on that quality assessment dimension (the heading angle of the specified object in the image).

[0212] The percentage of a specified object's area in an image represents the proportion of the image area occupied by that specified object within the total image area. It's understood that the image areas outside the specified object's area typically represent invalid information (such as background information). Therefore, a larger percentage of the specified object's area indicates a larger proportion of valid information in the image; conversely, a smaller percentage indicates a smaller proportion of valid information.

[0213] Correspondingly, if the area of ​​a specified object in an image occupies a large proportion, the image will have a high sub-score on that quality assessment dimension (the proportion of the area of ​​a specified object in the image); if the area of ​​a specified object in an image occupies a small proportion, the image will have a low sub-score on that quality assessment dimension (the proportion of the area of ​​a specified object in the image).

[0214] In this way, the quality assessment model can evaluate the image quality of the input image from multiple quality assessment dimensions, that is, output the image as a sub-score on each quality assessment dimension. This avoids evaluating image quality from a single dimension, improving the accuracy of image quality assessment. Consequently, it ensures the image quality of the images input into the database.

[0215] After obtaining the predicted sub-scores of the image to be screened on each quality evaluation dimension, the electronic device can calculate the total predicted score of the image to be screened based on each predicted sub-score and the current weights of each quality evaluation dimension represented by each branch network. The current weights of each quality evaluation dimension are obtained by adjusting the initial values ​​of the weights of each quality evaluation dimension. The specific process of adjusting the weights of each quality evaluation dimension will be explained in subsequent embodiments.

[0216] The current weight range of each quality evaluation dimension is [0,1], that is, the total predicted score of the image to be screened is positively correlated with the predicted sub-score of the image on each quality evaluation dimension.

[0217] In some embodiments, the electronic device may calculate a weighted sum of the prediction sub-scores based on the current weights of each prediction sub-score and each quality assessment dimension, as the total predicted score for the image to be filtered.

[0218] Understandably, when calculating the total predicted score for an image using a weighted sum method, the influence of a sub-score on a single quality assessment dimension on the total predicted score is limited. In other words, if an image has a low quality score only on one quality assessment dimension, its sub-score on that dimension will also be low. However, the total predicted score may be affected by sub-scores on other quality assessment dimensions, potentially failing to reflect this. Understandably, adding such an image to the image library is not appropriate in this situation. If the judgment is based solely on the total predicted score, the electronic device might add the image to the library, leading to lower detection accuracy when subsequently performing detection based on images in the library.

[0219] In some embodiments, the step of calculating the total predicted score of the image to be screened based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network includes:

[0220] Based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network, the total predicted score of the image to be screened is calculated according to the second formula.

[0221] The second formula is:

[0222] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0223] f(z) represents the predicted total score of the image to be selected; k represents the number of each quality assessment dimension; f(x1) represents the predicted sub-score of the image to be selected in the first quality assessment dimension, and α represents the current weight of the first quality assessment dimension; f(x i f(x) represents the predicted sub-rating of the image to be filtered in the i-th quality assessment dimension, and β represents the current weight of the i-th quality assessment dimension; k) represents the predicted sub-score of the image to be filtered in the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0224] In this embodiment, after obtaining the predicted sub-scores of the image to be screened on each quality evaluation dimension, the electronic device can calculate the product of the predicted sub-score of the image to be screened on that quality evaluation dimension and the current weight of that quality evaluation dimension (which can be called the first product) for each quality evaluation dimension. Then, it calculates the product of the first products corresponding to each quality evaluation dimension (which can be called the second product). Next, it calculates the product of a preset first parameter (i.e., w) and the second product (which can be called the third product), and uses the sum of the third product and the preset second parameter (i.e., n) as the total predicted score of the image to be screened.

[0225] The preset first parameter value (w) is greater than 0, meaning that the predicted total score of the image to be screened is positively correlated with the second product.

[0226] For example, the range of n can be [0,1].

[0227] Correspondingly, the method described above for calculating the predicted total score of the image can be called the multiplicative fusion method.

[0228] In one implementation, if the quality evaluation dimensions include: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and the area proportion of a specified object in the image, then the second formula can be expressed as follows during the calculation of the predicted total score of the current image to be screened using the multiplicative fusion method:

[0229] f(z)=w×(αf(x1)×βf(x2)×γf(x3)×δf(x4)×εf(x5))+n;

[0230] Where f(z) represents the total predicted score of the image to be filtered; f(x1) represents the predicted sub-score of the image to be filtered in terms of image sharpness, α represents the current weight of the first quality evaluation dimension (image sharpness); f(x2) represents the predicted sub-score of the image to be filtered in terms of image visibility, β represents the current weight of the second quality evaluation dimension (image visibility); f(x3) represents the predicted sub-score of the image to be filtered in terms of the pitch angle of a specified object in the image, γ represents the current weight of the third quality evaluation dimension (pitch angle of a specified object in the image); f(x4) represents the predicted sub-score of the image to be filtered in terms of the heading angle of a specified object in the image, δ represents the current weight of the fourth quality evaluation dimension (heading angle of a specified object in the image); f(x5) represents the predicted sub-score of the image to be filtered in terms of the area proportion of a specified object in the image, ε represents the current weight of the fifth quality evaluation dimension (area proportion of a specified object in the image); w is a preset first parameter, and n is a preset second parameter.

[0231] Understandably, when calculating the predicted total score for an image using multiplicative fusion, a low sub-score on a single quality assessment dimension will result in a low predicted total score. In other words, if an image has a low quality score only on one quality assessment dimension, its sub-score on that dimension will also be low, leading to a low predicted total score. In this case, the electronic device can determine that the image is unsuitable for addition to its image library based on its predicted total score.

[0232] Based on the above processing, when calculating the predicted total score of the current image to be screened using the multiplication fusion method, any low predicted sub-score will lead to a significant decrease in the predicted total score. In other words, the predicted total score of the current image to be screened depends on the image's performance across all quality assessment dimensions, and defects in any quality assessment dimension will significantly affect the overall quality assessment result (i.e., the predicted total score).

[0233] That is, compared with calculating the predicted total score of an image by the weighted sum method described above, the multiplicative fusion method used in this embodiment of the application can ensure that the images added to the image library have high quality in each quality evaluation dimension, that is, the overall quality of the images is high. Therefore, when performing detection based on the images in the image library, the accuracy of the recognition results is improved.

[0234] Regarding step S103, after obtaining the predicted total score of the current image to be filtered, the electronic device can determine whether the predicted total score of the current image to be filtered is less than a preset filtering threshold. For example, the preset filtering threshold can be 0.1.

[0235] If the total predicted score of the current image to be screened is less than the preset filtering threshold, it indicates that the overall quality of the current image to be screened is low. Accordingly, the current image to be screened is not suitable for subsequent detection processes. Therefore, the electronic device can determine that the current image to be screened will not be retained.

[0236] If the predicted total score of the current image to be screened is not less than the preset filtering threshold, it indicates that the overall quality of the current image to be screened is high. Accordingly, the current image to be screened can be used for subsequent detection processes. Therefore, the electronic device can determine that the screening result of the current image to be screened is to be retained.

[0237] In one implementation, the preset range can be an adjacent interval of a specified size, centered on a preset filtering threshold. For example, when the preset filtering threshold is 0.1, the preset range can be [0.05, 0.2].

[0238] Understandably, if the predicted total score of the current image to be filtered falls within a preset range, that is, if the predicted total score of the current image to be filtered is close to the preset filtering threshold, then the current image to be filtered is neither an image of very high quality that must be retained, nor an image of very low quality that must be excluded. In other words, the filtering result for the current image to be filtered falls on the borderline between retention and exclusion. Correspondingly, if the current filtering ratio is lower than the target filtering ratio, it indicates that the proportion of images to be excluded is relatively small; therefore, the filtering intensity can be increased, and the current image to be filtered can be excluded. If the current filtering ratio is higher than the target filtering ratio, it indicates that the proportion of images to be excluded is relatively large; therefore, the filtering intensity can be decreased, and the current image to be filtered can be retained.

[0239] At this point, the electronic device can adjust the current weights of each quality evaluation dimension based on the current filtering ratio and the target filtering ratio, using a reinforcement learning algorithm. The specific process of adjusting the weights of each quality evaluation dimension will be described in subsequent embodiments.

[0240] In this application, performing steps S101 to S103 once can be referred to as a scoring and filtering process.

[0241] Understandably, if the total number of images to be filtered is fixed, the number of images that have not yet been filtered will decrease by one after each execution of the process of determining the image filtering results.

[0242] In other words, during the Nth execution of the scoring and filtering process (hereinafter referred to as the Nth execution process), after the electronic device determines the filtering result of the current image to be filtered, it can return to execute the above step S101, that is, start executing the N+1th scoring and filtering process. It can be understood that the images that have not been filtered in the N+1th execution process are different from the images that have not been filtered in the Nth execution process, lacking the images that were currently to be filtered in the Nth execution process.

[0243] Accordingly, the electronic device can perform the above scoring and filtering process multiple times until the filtering results of all images to be filtered are obtained.

[0244] Regarding step S104, the image library refers to the storage space used to store images. After obtaining the filtering results of all images to be filtered, the electronic device can obtain images to be added to the current image library based on the images selected as the retained images.

[0245] For example, in the above identity authentication scenario, after obtaining the photos of all persons authorized to enter the designated place (i.e. all images to be screened) uploaded by the user, the electronic device can obtain the images to be added to the current image library according to the above steps S101 to S104, and then add the images to the image library.

[0246] In some embodiments, see Figure 2, which is a second flowchart of the image filtering method provided in the embodiments of this application.

[0247] Based on Figure 1, step S104 includes:

[0248] S1041: Select one currently unprocessed image from the images that are selected as the images to be processed.

[0249] S1042: Extract features from the current image to be processed to obtain the current first image features.

[0250] S1043: Based on the matching results of each of the current first feature clusters and the current first image features, determine whether the current image to be processed contains the specified object.

[0251] Among them, each of the current first feature clusters is obtained by clustering the image features of the images contained in the current first image set; the images in the current first image set do not contain the specified object.

[0252] S1044: If the image to be processed contains the specified object, then the image to be processed is determined to be the image to be added to the current image library.

[0253] If there are currently unprocessed images, return to step S1041.

[0254] In this embodiment, the electronic device can select an unprocessed image from the images whose filtering results are retained, according to the order in which the filtering results are determined, as the current image to be processed, and process it according to steps S1042 to S1044. Alternatively, the electronic device can randomly select one image from the currently unprocessed images whose filtering results are retained, as the current image to be processed, and process it according to steps S1042 to S1044. The specific method of determining the currently unprocessed image is not limited here.

[0255] In this application, performing steps S1041 to S1044 once can be referred to as one modeling and filtering process. Accordingly, after each execution of the modeling and filtering process, the number of currently unprocessed images will decrease by one.

[0256] In other words, during the Nth execution of modeling filtering, after the electronic device determines whether the current image to be processed contains the specified object, it can return to execute the above step S1041, that is, start the N+1th scoring filtering process. It can be understood that the currently unprocessed image in the N+1th execution process, compared to the currently unprocessed image in the Nth execution process, lacks the image to be processed in the Nth execution process.

[0257] For steps S1042 to S1043, the electronic device can use a preset feature extraction network to extract features from the image to be processed, thereby obtaining the image features of the image to be processed (i.e., the current first image features). For example, the preset feature extraction network can be a Transform backbone network.

[0258] The current first feature clusters are obtained by clustering the image features of the images contained in the current first image set. The images in the current first image set do not contain the specified object.

[0259] In one implementation, the electronic device can extract features from each image in the current first image set to obtain the image features of each image in the current first image set. For example, the image features of each image in the current first image set can be represented as: F enroll ={f e1 ,f e2 ,f e3 …,f em}. Among them, f e1 f represents the image feature of the first image in the current first image set. e2 f represents the image features of the second image in the current first image set.e3 f represents the image feature of the third image in the current first image set, and so on. em F represents the image feature of the m-th image in the current first image set; enroll It represents the set of image features of all images in the current first image set that have been recorded (enrolled).

[0260] Furthermore, the electronic device can cluster the image features of each image in the current first image set based on a preset clustering algorithm, obtaining multiple feature clusters (i.e., the current first feature clusters). For example, the preset clustering algorithm can be a GCN (Graph Convolutional Networks) clustering algorithm. Correspondingly, the current first feature clusters can be represented as:

[0261] GCNF enroll ={{f X1 ,f X2 ,f X3 ,…}1,{f Y1 ,f Y2 ,f Y3 ,…}2,…,{f Z1 ,f Z2 ,f Z3 ,…} n};

[0262] Among them, {f X1 ,f X2 ,f X3 …,}1 represents the first feature cluster in the current first feature cluster, f X1 f represents the first image feature in the first feature cluster. X2 f represents the second image feature in the first feature cluster. X3 This represents the third image feature in the first feature cluster, and so on. The number of image features contained in the first feature cluster is not limited here. Y1 ,f Y2 ,f Y3 ,…}2 represents the second feature cluster in each of the current first feature clusters, f Y1 f represents the first image feature in the second feature cluster. Y2 f represents the second image feature in the second feature cluster. Y3 This represents the third image feature in the second feature cluster, and so on. The number of image features contained in the second feature cluster is not limited here. Z1 ,f Z2 ,f Z3 ,…} nf represents the nth feature cluster among the current first feature clusters. Z1 f represents the first image feature in the nth feature cluster. Z2 f represents the second image feature in the nth feature cluster. Z3 This represents the third image feature in the nth feature cluster, and so on. The number of image features contained in the nth feature cluster is not limited here. GCNF enroll This represents the set of the currently enrolled first feature clusters.

[0263] Accordingly, for each current first feature cluster, the electronic device can calculate the mean of the image features contained in that first feature cluster, which serves as the cluster center (or class center) of that first feature cluster. For example, the cluster center of the j-th feature cluster in the current set of first feature clusters can be calculated using the following formula:

[0264]

[0265] Among them, U j Let f represent the cluster center of the j-th feature cluster in the current first feature cluster, k represent the number of image features contained in the j-th feature cluster, and f i This represents the i-th image feature in the j-th feature cluster.

[0266] Furthermore, during the process of performing a modeling and filtering, after obtaining the current first image features, the electronic device can determine whether the current image to be processed contains a specified object based on the matching results between the current first feature clusters and the current first image features.

[0267] In some embodiments, during the process of determining whether a specified object is contained in the current image to be processed based on the matching results of the current first feature clusters and the current first image features, the electronic device may utilize the similarity between image features to determine whether the current image to be processed contains a specified object. Referring to Figure 3, Figure 3 is a flowchart of a method for determining whether a specified object is contained in the current image to be processed according to an embodiment of this application, including the following steps:

[0268] S301: Calculate the first similarity between the current first image feature and the cluster centers of the current first feature clusters.

[0269] S302: Select the first feature cluster with the highest first similarity from the current first feature clusters.

[0270] S303: Calculate the second similarity between the current first image feature and each image feature contained in the currently selected first feature cluster.

[0271] S304: Determine whether the specified object is contained in the current image to be processed based on the second similarity scores corresponding to the current image to be processed.

[0272] In this embodiment, for each current first feature cluster, the electronic device can calculate the cosine similarity (i.e., the first similarity) between the current first image feature and the cluster center of the first feature cluster, thus obtaining the first similarity corresponding to the current first image feature. For example, the first similarity corresponding to the current first image feature can be expressed as: S Result ={cos u1α cos u2α cos u3α ,…,cos unα}

[0273] Among them, S Result This represents the set of first similarities corresponding to the current first image feature; cos u1α Cosine similarity between the current first image feature and the cluster center of the first feature cluster in each of the current first feature clusters; u2α Cosine similarity between the current first image feature and the cluster center of the second feature cluster in each of the current first feature clusters; u3α Cosine similarity is represented by the cosine similarity between the current first image feature and the cluster center of the third feature cluster in each of the current first feature clusters, and so on. unα This represents the cosine similarity between the current first image feature and the cluster center of the nth feature cluster in each of the current first feature clusters.

[0274] Furthermore, the electronic device can select the first feature cluster with the highest first similarity from the current first feature clusters (which can be called the current fourth feature cluster). For example, the first similarity corresponding to the current fourth feature cluster can be expressed as:

[0275] cos targetα =max{S Result};

[0276] Where, max{S Result} represents the maximum value among the first similarities corresponding to the current first image feature, cos targetα This represents the first similarity corresponding to the current fourth feature cluster.

[0277] It is understandable that the current fourth feature cluster is the one with the highest similarity among all the current first feature clusters. In other words, the current fourth feature cluster is the feature cluster that is most similar to the current first image feature among all the current first feature clusters.

[0278] Therefore, after determining the current fourth feature cluster, the electronic device can further utilize the image features contained in the current fourth feature cluster to determine whether the image to be processed contains the specified object.

[0279] Accordingly, the electronic device can use the DAM (Difference Alignment Index) algorithm to obtain the second similarity scores corresponding to the current image to be processed. That is, for each image feature contained in the current fourth feature cluster, the electronic device can calculate the cosine similarity (i.e., the second similarity score) between the current first image feature and that image feature, and determine whether the current image to be processed contains the specified object based on the second similarity scores corresponding to the current image to be processed.

[0280] In some embodiments, step S304 includes:

[0281] Step 1: Determine the largest number of second similarities among the first preset number of second similarities corresponding to the current image to be processed.

[0282] Step 2: If the first preset number of largest second similarities are all greater than the first preset similarity threshold, then it is determined that the image to be processed does not contain the specified object.

[0283] Step 3: Otherwise, determine if the specified object is contained in the current image to be processed.

[0284] In this embodiment of the application, after obtaining each second similarity corresponding to the image to be processed, the electronic device can determine the first preset number of the largest second similarities among the second similarities corresponding to the image to be processed.

[0285] Furthermore, since the image features in the current fourth feature cluster are those that do not contain the specified object, if the first preset number of largest second similarities are all greater than the first preset similarity threshold, it indicates that the image features of the image to be processed are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the image to be processed does not contain the specified object.

[0286] Otherwise, if the largest number of second similarities in the first preset number are not all greater than the first preset similarity threshold, it indicates that the image features of the image to be processed are not similar to the image features of an image that does not contain the specified object. Therefore, it can be determined that the image to be processed contains the specified object.

[0287] In one implementation, to facilitate the comparison of similarity scores, after calculating the cosine similarity between the current first image feature and each image feature contained in the current fourth feature cluster, the cosine similarity scores can be normalized. That is, the values ​​of each cosine similarity score are mapped to [0,1] to obtain the second similarity scores between the current first image feature and each image feature contained in the current fourth feature cluster.

[0288] Thus, after determining the feature cluster closest to the current first image feature (i.e., the current fourth feature cluster), the image features contained in the current fourth feature cluster can be further utilized to determine whether the current image to be processed contains the specified object. That is, determining whether the current image to be processed contains the specified object based on the first preset number of largest second similarities can reduce the amount of computation and improve the efficiency of image screening.

[0289] Based on the above processing, the current first image feature is first compared with the cluster centers of each feature cluster to determine the feature cluster closest to the current first image feature (i.e., the current fourth feature cluster). Then, the current first image feature is compared with each image feature contained in the current fourth feature cluster. In this way, compared with the method of directly comparing with each image feature, the amount of computation can be reduced, and the image features of the recorded image can be used to quickly identify the image to be processed.

[0290] For step S1044, if the image to be processed contains a specified object, then the image to be processed is determined to be an image to be added to the current image library, and the process returns to step S1041.

[0291] In some embodiments, the method further includes:

[0292] If it is determined that the specified object is not contained in the current image to be processed, the current image to be processed is added to the current first image set.

[0293] In this embodiment of the application, if it is determined that the image to be processed does not contain the specified object, the electronic device can add the image to be processed to the current first image set. In the subsequent process of performing the above modeling and filtering, each feature cluster can be determined according to the new current first image set.

[0294] In this way, the first image set can be dynamically updated. That is, the images (i.e., negative samples) that do not contain the specified object recorded in the first image set can be continuously updated during the image filtering process. In other words, the images that do not contain the specified object obtained during the image filtering process can be retained and participate in the subsequent filtering process. This can improve the accuracy of subsequent image filtering and achieve continuous optimization of filtering precision.

[0295] Based on the above processing, for each image to be added to the image library, it can first be identified as the current image to be filtered, and the scoring filtering process described above can be performed. If the image is selected for retention, it is identified as the current image to be processed, and the modeling filtering process described above can be performed. If it is determined that the image to be processed contains a specified object, it is determined as the image to be added to the current image library. In this way, by combining the scoring filtering and modeling filtering methods, multi-level filtering of the image can be performed, thereby improving the accuracy of image selection and enhancing the quality of images stored in the image library.

[0296] In some embodiments, for each image to be added to the image library, the above-described modeling filtering process can be performed first, and if it is determined that the current image to be processed contains a specified object, then the above-described scoring filtering process can be performed. Accordingly, see Figure 4, which is a third flowchart of the image filtering method provided in the embodiments of this application. Based on Figure 1, before step S102, the method further includes:

[0297] S105: Extract features from the current image to be filtered to obtain the current second image features.

[0298] S106: Based on the matching results of each of the current second feature clusters and the current second image features, determine whether the current image to be filtered contains the specified object.

[0299] The current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; the images in the current second image set do not contain the specified object.

[0300] Step S102 includes:

[0301] S1021: If the image to be screened contains a specified object, input the image to be screened into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the total predicted score of the image to be screened based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network.

[0302] In this embodiment of the application, the process of extracting features from the current image to be screened to obtain the current second image features can refer to the process of obtaining the current first image features in step S1042 above, and will not be repeated here.

[0303] The process of obtaining the current second feature clusters can be referred to the process of obtaining the current first feature clusters in the above embodiments, and will not be repeated here. The description of the current second image set can be referred to the description of the current first image set in the above embodiments, and will not be repeated here.

[0304] The process of determining whether the specified object is contained in the current image to be filtered based on the matching results of each of the current second feature clusters and the current second image features can be referred to in step S1043 above, which is the process of determining whether the specified object is contained in the current image to be processed based on the matching results of each of the current first feature clusters and the current first image features. It will not be elaborated here.

[0305] In some embodiments, step S106 includes:

[0306] Step 1: Calculate the third similarity between the current second image feature and the cluster centers of each of the current second feature clusters.

[0307] Step 2: Select the second feature cluster with the highest third similarity from the current second feature clusters.

[0308] Step 3: Calculate the fourth similarity between the current second image feature and each image feature contained in the currently selected second feature cluster.

[0309] Step 4: Based on the fourth similarity scores corresponding to the current image to be filtered, determine whether the current image to be filtered contains the specified object.

[0310] In this embodiment of the application, the process of calculating the third similarity corresponding to the current second image feature can refer to the process of calculating the first similarity corresponding to the current first image feature in step S301 above, and will not be repeated here.

[0311] Furthermore, the electronic device can select the second feature cluster with the highest third similarity from the current second feature clusters (which can be called the current fifth feature cluster).

[0312] It is understandable that the current fifth feature cluster is the one with the highest corresponding third similarity among all the current second feature clusters. In other words, the current fifth feature cluster is the feature cluster that is most similar to the current second image feature among all the current second feature clusters.

[0313] Therefore, after determining the current fifth feature cluster, the electronic device can further utilize the image features contained in the current fifth feature cluster to determine whether the specified object is contained in the image to be processed. That is, it calculates the fourth similarity between the current second image features and the image features contained in the current fifth feature cluster (i.e., the currently selected second feature cluster).

[0314] The process of determining whether the image to be filtered contains a specified object based on the fourth similarity scores of the image to be filtered can be referred to in step S304 above, which is the process of determining whether the image to be processed contains a specified object based on the second similarity scores of the image to be processed. It will not be elaborated here.

[0315] In one implementation, step four above includes:

[0316] Determine the largest fourth similarity among all fourth similarities corresponding to the current image to be filtered, based on a second preset number of values. If all of these largest fourth similarities are greater than a second preset similarity threshold, then the current image to be filtered does not contain the specified object; otherwise, the current image to be filtered contains the specified object.

[0317] In this embodiment of the application, the process of determining the second preset number of largest fourth similarities among the fourth similarities corresponding to the current image to be screened can refer to the process of determining the first preset number of largest second similarities among the second similarities corresponding to the current image to be processed in step 1 above.

[0318] Furthermore, since the image features in the current fifth feature cluster are those that do not contain the specified object, if all of the second preset number of fourth similarities are greater than the second preset similarity threshold, it indicates that the image features of the image to be filtered are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the image to be filtered does not contain the specified object.

[0319] Otherwise, if the second preset number of fourth similarities are not all greater than the second preset similarity threshold, it indicates that the image features of the image to be filtered are not similar to the image features of an image that does not contain the specified object. Therefore, it can be determined that the image to be filtered contains the specified object.

[0320] Thus, after determining the feature cluster closest to the current second image feature (i.e., the current fifth feature cluster), the image features contained in the current fifth feature cluster can be further utilized to determine whether the current image to be processed contains the specified object. That is, based on the first second preset number of fourth similarities with larger values, determining whether the current image to be filtered contains the specified object can reduce the amount of computation and improve the efficiency of image filtering.

[0321] Based on the above processing, the current second image feature is first compared with the cluster centers of each feature cluster to determine the feature cluster closest to the current second image feature (i.e., the current fifth feature cluster). Then, the current second image feature is compared with each image feature contained in the current fifth feature cluster. In this way, compared with the method of directly comparing with each image feature, the amount of computation can be reduced, and the image features of the recorded image can be used to quickly identify the image to be screened.

[0322] Correspondingly, if the image to be processed contains a specified object, a scoring filtering process can be performed. That is, if the image to be filtered contains a specified object, the image to be filtered is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model. Then, the electronic device can calculate the predicted total score of the image to be filtered based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network.

[0323] In some embodiments, the method further includes:

[0324] If it is determined that the specified object is not contained in the current image to be filtered, the current image to be filtered is added to the current second image set, and the process returns to step S101.

[0325] In this embodiment of the application, if it is determined that the image to be filtered does not contain the specified object, the electronic device can add the image to be filtered to the current second image set. In the subsequent process of performing the above modeling and filtering, each feature cluster can be determined according to the new current second image set.

[0326] In this way, the second image set can be dynamically updated. That is, the images that do not contain the specified object (i.e., negative samples) recorded in the second image set can be continuously updated during the image filtering process. In other words, the images that do not contain the specified object obtained during the image filtering process can be retained and participate in the subsequent filtering process. This can improve the accuracy of subsequent image filtering and achieve continuous optimization of filtering precision.

[0327] Based on the above processing, for each image to be added to the image library, it can first be identified as the current image to be filtered, and the modeling filtering process described above can be executed. If it is determined that the current image to be filtered contains the specified object, then the scoring filtering process described above can be executed. If the image is selected for retention, it is determined as the image to be added to the current image library. In this way, by combining the scoring filtering and modeling filtering methods, multi-level filtering of the image can be performed, thereby improving the accuracy of image filtering and enhancing the quality of images stored in the image library.

[0328] In some embodiments, the electronic device can adjust the current weights of each quality evaluation dimension based on a reinforcement learning algorithm, according to the current filtering ratio and the target filtering ratio. Referring to Figure 5, which is a flowchart of adjusting the current weights of each quality evaluation dimension according to an embodiment of this application, the process includes the following steps:

[0329] S501: Get the current state.

[0330] The current state includes: the current filtering ratio, the number of images whose filtering results have been determined, and the current weight of each quality evaluation dimension.

[0331] S502: Determine the expected gain when taking each alternative action in the current state, as the first expectation for the current state.

[0332] One of the alternative actions includes adjusting the weights of each quality evaluation dimension.

[0333] S503: Select the alternative action corresponding to the highest current first expectation as the action to be taken now.

[0334] S504: Adjust the current weights of each quality assessment dimension based on the actions required to be taken.

[0335] In this embodiment of the application, the current state includes: the current filtering ratio, the number of images for which the current filtering results have been determined, and the current weight of each quality evaluation dimension.

[0336] For example, if the above quality evaluation dimensions include: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image, the current state can be represented as: S=(n,p,α,β,γ,δ,ε).

[0337] Where S represents the current state, n represents the number of images whose filtering results have been determined, p represents the current filtering ratio, α represents the current weight of image sharpness, β represents the current weight of image visibility, γ represents the current weight of the pitch angle of the specified object in the image, δ represents the current weight of the heading angle of the specified object in the image, and ε represents the current weight of the area proportion of the specified object in the image.

[0338] The current filtering ratio is the ratio of the number of images that are excluded from the filter to the number of images whose filtering results have been determined. For example, the number of images excluded from the filter can be denoted as n1, and the number of images retained can be denoted as n2. Correspondingly, the number of images whose filtering results have been determined (n) is the sum of n1 and n2. The current filtering ratio p can be expressed as:

[0339] An alternative action includes the adjustment of the weights of each quality assessment dimension. For example, if there are m alternative actions, the first alternative action can be denoted as A1, the second alternative action can be denoted as A2, and so on, with the m-th alternative action being denoted as An.

[0340] Accordingly, the electronic device determines the expected gain when taking each alternative action in the current state, as the current first expectation. For example, the current first expectation includes: Q(S,A1), Q(S,A2), ..., Q(S,Am). Here, Q(S,A1) represents the expected gain when taking the first alternative action (A1) in the current state, Q(S,A2) represents the expected gain when taking the second alternative action (A2) in the current state, and so on, Q(S,Am) represents the expected gain when taking the nth alternative action (Am) in the current state.

[0341] After obtaining the current first expectation, the electronic device can select the alternative action corresponding to the highest current first expectation as the action to be taken. Here, the action to be taken can be denoted as A, and correspondingly, the action to be taken can be expressed as: A=(Δα,Δβ,Δγ,Δδ,Δε).

[0342] Where Δα represents the weight adjustment of image sharpness, Δβ represents the weight adjustment of image visibility, Δγ represents the weight adjustment of the pitch angle of a specified object in the image, Δδ represents the weight adjustment of the heading angle of a specified object in the image, and Δε represents the weight adjustment of the area proportion of a specified object in the image.

[0343] Furthermore, the electronic device can adjust the current weights of each quality assessment dimension based on the action to be taken. Specifically, it calculates the sum of the current weight (α) of image sharpness and the adjustment amount (Δα) of the image sharpness weight; the sum of the current weight (β) of image visibility and the adjustment amount (Δβ) of the image visibility weight; the sum of the current weight (γ) of the pitch angle of a specified object in the image and the adjustment amount (Δγ) of the pitch angle weight of the specified object in the image; the sum of the current weight (δ) of the heading angle of a specified object in the image and the adjustment amount (Δδ) of the heading angle weight of the specified object in the image; and the sum of the current weight (ε) of the area proportion of a specified object in the image and the adjustment amount (Δε) of the area proportion weight of the specified object in the image. In this way, the current weights of each quality assessment dimension are updated.

[0344] During the execution of steps S102 to S103 for the image to be filtered, after the electronic device determines the filtering result of the image to be filtered, the current state (S) will change the current filtering ratio (p). Accordingly, if p in the current state (S) does not reach the target filtering ratio, the electronic device can update the current weights of each quality evaluation dimension according to steps S501 to S504. If p in the current state (S) reaches the target filtering ratio, the current state is considered to have reached the target state (which can be called Target S), and at this time, there is no need to update the current weights of each quality evaluation dimension.

[0345] Understandably, after returning to step S101 and determining the next image to be filtered, the electronic device can re-determine whether to update the current weights of each quality evaluation dimension based on the current state, until the filtering results of all images to be filtered are determined.

[0346] Based on the above processing, during the image filtering process, the target filtering ratio and the current filtering ratio can be further combined, and the current weights of each quality evaluation dimension can be dynamically adjusted using a reinforcement learning algorithm. In subsequent image filtering processes, the predicted total score of the image is determined based on the updated weights of each quality evaluation dimension, thereby determining the image filtering result. In other words, at each stage of filtering all images to be filtered, the current filtering ratio can be continuously adjusted towards the target filtering ratio, improving the flexibility of image filtering and meeting the personalized needs of users.

[0347] Furthermore, it's understandable that the weights of each quality evaluation dimension are adjusted only if the predicted total score of the image to be screened falls within a preset range. This ensures that the predicted total score of images with higher total scores (i.e., better image quality) is unaffected by weight adjustments, thus guaranteeing that correctly retained images are not affected. Conversely, the predicted total score of images with lower total scores (i.e., poorer image quality) is unaffected by weight adjustments, ensuring that correctly rejected images are not affected.

[0348] In some embodiments, the method further includes:

[0349] After determining the action that needs to be taken, the current first expectation is updated based on the first formula.

[0350] The first formula is:

[0351] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0352]

[0353] Q(S, A) represents the current first expectation, S represents the current state, and A represents the current action to be taken; Q′(S, A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, and θ represents the discount factor; R represents the reward; S′ represents the next state after obtaining the screening result of the current image to be screened, maxQ(S′, A′) represents the maximum value among the second expectations of the benefits obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value of the second expectation;

[0354] s.t. represents the constraint condition, and R(S, A) represents that the reward R is a function based on the current state S and the current action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c represents the preset filtering threshold, R min represents the minimum value in the preset range, R max represents the maximum value in the preset range, and R min <R c <R max ; M represents the target filtering ratio, p represents the current filtering ratio; otherwise represents other cases.

[0355] In the embodiment of the present application, after determining the current action to be taken, the electronic device can calculate the reward R according to the incentive function R(S, A) in the above first formula and update the current first expectation.

[0356] Among them, if the predicted total score f(z) of the current image to be screened satisfies f(z) < R c and (and) R min < f(z) < R max , and the current filtering ratio p satisfies p < M, then the value of the reward R is 1; if the predicted total score f(z) of the current image to be screened satisfies f(z) > R max , or the current filtering ratio p satisfies p ≥ M, then the value of the reward R is -1; otherwise represents that the predicted total score f(z) of the current image to be screened and the current filtering ratio p do not satisfy any of the above situations, and at this time, the value of the reward R is 0.

[0357] Based on the above processing, in the process of updating the current first expectation, the electronic device can set the reward function according to the relationship between the current filtering ratio and the target filtering ratio. That is, if the current filtering ratio is less than the target filtering ratio, a positive reward is given, that is, the value of R is 1; if the current filtering ratio is not less than the target filtering ratio, a negative reward is given, that is, the value of R is -1.

[0358] In some embodiments, referring to Figure 6, Figure 6 is a flowchart of a training quality assessment model provided by an embodiment of this application. The quality assessment model is trained through the following steps:

[0359] S601: Get a sample image containing the specified object.

[0360] The labels of a sample image include: the sample sub-scores of the sample image on each quality assessment dimension, and the total sample score of the sample image; the total sample score of a sample image is obtained based on the initial values ​​of the weights of each sample sub-score and each quality assessment dimension of the sample image.

[0361] S602: For each sample image, input the sample image into the quality assessment model of the initial structure to obtain the training sub-scores output by each branch network in the quality assessment model of the initial structure, and calculate the total training score of the sample image based on the initial values ​​of the weights of each quality assessment dimension represented by each branch network and the training sub-scores.

[0362] S603: Calculate the loss value based on the training sub-scores and total training score of the sample image, as well as the sample sub-scores and total sample score of the sample image.

[0363] S604: Adjust the model parameters of the initial structure quality assessment model based on the obtained loss value until convergence is achieved.

[0364] In this embodiment of the application, the sample image can be an image from a publicly available image set.

[0365] It is understandable that the specified object in the image filtering process is of the same type as the specified object in the sample image. That is, if the specified object in the image to be filtered is a human face, then the specified object in the sample image is also a human face.

[0366] The quality assessment model consists of a backbone network and multiple fully-connected layers (FC) (i.e., branch networks in this application). The backbone network can be a convolutional neural network (CNN), and each fully-connected layer corresponds to one quality assessment dimension.

[0367] Accordingly, after obtaining an image, the quality assessment model can input the image into a backbone network (CNN), and then input the data output by the backbone network into each fully connected layer. Each fully connected layer can process the input data. Furthermore, for each fully connected layer, the output of the fully connected layer can be processed based on the activation function corresponding to that fully connected layer to obtain a sub-score for the image on the quality assessment dimension represented by that fully connected layer.

[0368] Accordingly, for each sample image, the sample image is input into the quality assessment model of the initial structure to obtain the training sub-scores output by each branch network in the quality assessment model of the initial structure.

[0369] The process of calculating the total training score of the sample image based on the initial values ​​of the weights of each quality evaluation dimension represented by each training sub-score and each branch network can be referred to in step S102 above, which calculates the total predicted score of the image to be screened based on the current weights of each prediction sub-score and each quality evaluation dimension represented by each branch network. It will not be elaborated here.

[0370] The labels for a sample image include: the sample sub-scores for each quality assessment dimension, and the total sample score for the sample image.

[0371] The range of sample sub-scores for each quality evaluation dimension is preset according to actual needs. For example, the range of each sample sub-score can be [0, 100].

[0372] In one implementation, during the labeling process of a sample image, the sample sub-score of the sample image in each quality evaluation dimension can be determined by comparing it with another reference sample image in each quality evaluation dimension.

[0373] For example, if the quality evaluation dimensions include: image sharpness, image visibility, and the proportion of a specified object in the image, and the reference sample image has a sample sub-score of 50 for image sharpness, 50 for image visibility, and 50 for the proportion of a specified object in the image, then if a sample image (which can be called sample image 1) has higher sharpness than the reference sample image, its sample sub-score for image sharpness can be set to be greater than 50 (e.g., 60); if sample image 1 also has higher visibility than the reference sample image, its sample sub-score for image visibility can be set to be greater than 50 (e.g., 70); if sample image 1 has a smaller proportion of a specified object in the image compared to the reference sample image, its sample sub-score for image visibility can be set to be less than 50 (e.g., 40).

[0374] Accordingly, after determining the sample sub-scores of a sample image on each quality evaluation dimension, the total sample score of the sample image can be calculated based on the initial values ​​of the sample sub-scores and the weights of each quality evaluation dimension represented by each branch network. For example, the multiplication fusion method in the above embodiment can be referred to, which will not be elaborated here.

[0375] Accordingly, for a quality assessment dimension, the electronic device can calculate the loss value of the sample image in that quality assessment dimension based on the difference between the training sub-rating of the sample image in that dimension and the sample sub-rating of the sample image in that dimension. Furthermore, it can obtain the sub-rating loss value of the sample image in each quality assessment dimension.

[0376] Furthermore, for a given sample image, the electronic device can calculate the total score loss value based on the difference between the total training score of that sample image and the total sample score of that sample image.

[0377] Furthermore, the electronic device can calculate the weighted sum of the sub-rating loss value and the total rating loss value corresponding to the sample image, and adjust the model parameters of the quality assessment model of the initial structure based on the weighted sum until convergence is achieved.

[0378] Based on the above processing, the image screening method provided in this application can directly evaluate the quality of images through a quality assessment model. That is, in the process of evaluating the quality of images, there is no need for the user to use an additional undamaged reference image as an evaluation basis. In other words, in this application, the method for evaluating image quality is No-Reference Image Quality Assessment (NR-IQA).

[0379] Based on the same inventive concept, embodiments of this application provide an image recognition method that can be applied to electronic devices.

[0380] Referring to Figure 7, which is a first flowchart of an image recognition method provided in an embodiment of this application, the method includes the following steps:

[0381] S701: Acquire the image to be recognized.

[0382] S702: Extract features from the image to be recognized to obtain the features of the image to be recognized.

[0383] S703: Based on the matching results between the features of the image to be identified and the image features of the images in the current image library, the recognition result of the image to be identified is obtained.

[0384] The images in the current image library are obtained based on the image filtering method provided in this application.

[0385] Based on the above processing, an unselected image can be chosen from all images to be screened as the current image to be screened, and this image is then input into a pre-trained quality assessment model. The quality assessment model contains multiple branch networks, each representing a quality assessment dimension. Correspondingly, the model outputs sub-scores (i.e., predicted sub-scores) for each quality assessment dimension of the current image to be screened. Then, based on each predicted sub-score and the current weights of each quality assessment dimension, the predicted total score of the current image to be screened is calculated. Based on the predicted total score, the screening result (keeping or not keeping) of the current image to be screened is determined. Similarly, the screening result for each image among all images to be screened can be obtained by following the above steps. In this way, automatic scoring and screening of each image to be screened can be achieved through a pre-trained quality assessment model, eliminating the need for manual screening and improving screening efficiency.

[0386] After obtaining the filtering results for each image to be filtered, since the filtering results are determined based on the predicted total score of the image, the predicted total score of the image can reflect the overall quality of the image across multiple quality evaluation dimensions. Therefore, based on the images selected for retention, images are obtained for addition to the current image library, ensuring that high-quality images are added to the current image library for easier subsequent processing. Furthermore, the image recognition method provided in this application embodiment can utilize the image features of images in the image library to detect the image to be recognized and obtain the recognition result of the image to be recognized. In this way, the accuracy of the recognition result of the image to be recognized can be guaranteed.

[0387] For steps S701 to S702, the image to be identified obtained by the electronic device represents an image in which a specified object needs to be detected.

[0388] For example, in the above identity authentication scenario, the image to be identified can be: an image of a person who needs to enter the specified location, captured by the gate at the entrance of the specified location.

[0389] Furthermore, the electronic device can utilize a pre-defined feature extraction network to extract features from the image to be recognized, thereby obtaining the image features (i.e., the features of the image to be recognized). For example, the pre-defined feature extraction network can be a Transform backbone network.

[0390] For step S703, the recognition result of the image to be recognized is obtained based on the matching result between the features of the image to be recognized and the image features of the images in the current image library.

[0391] In one implementation, the electronic device can calculate the similarity between the features of the image to be identified and the image features of each image in the current image library. Then, from the images in the current image library, the image with the highest similarity is determined as the target image. If the similarity of the target image is greater than a preset similarity threshold, then the identity of the specified object in the image to be identified is determined to be consistent with the identity of the specified object in the target image. That is, the identification result of the image to be identified indicates the identity of the specified object in the image to be identified.

[0392] In some embodiments, referring to FIG8, FIG8 is a second flowchart of the image recognition method provided in the embodiments of this application. Based on FIG7, before step S703, the method further includes:

[0393] S704: Calculate the fifth similarity between the features of the image to be identified and the cluster centers of the current third feature clusters.

[0394] Among them, each of the current third feature clusters is obtained by clustering the image features of the images contained in the current third image set; the images in the current third image set do not contain the specified object.

[0395] S705: Select the third feature cluster with the highest fifth similarity from the current third feature clusters.

[0396] S706: Calculate the sixth similarity between the features of the image to be identified and the features of each image contained in the currently selected third feature cluster.

[0397] S707: Determine whether the image to be identified contains the specified object based on the sixth similarity scores corresponding to the image to be identified.

[0398] Step S703 includes:

[0399] S7031: When the image to be identified contains a specified object, the identification result of the image to be identified is obtained based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0400] In this embodiment, the process of obtaining the current third feature clusters can be referred to the process of obtaining the current first feature clusters in the above embodiments, and will not be repeated here.

[0401] The current third image set can be either the first image set or the second image set in this application. Alternatively, the current third image set can also be a newly generated image set during the image recognition process.

[0402] The process of calculating the fifth similarity corresponding to each detected image feature can refer to the process of calculating the first similarity corresponding to the current first image feature in step S301 above, and will not be repeated here.

[0403] Furthermore, the electronic device can select the third feature cluster with the highest similarity from the current third feature clusters (which can be called the current sixth feature cluster).

[0404] It is understandable that the current sixth feature cluster is the fifth feature cluster with the highest similarity among all the current third feature clusters. In other words, the current sixth feature cluster is the feature cluster that is most similar to the features of the image to be identified among all the current third feature clusters.

[0405] Therefore, after determining the current sixth feature cluster, the electronic device can further utilize the image features contained in the current sixth feature cluster to determine whether the image to be identified contains a specified object. That is, it calculates the sixth similarity between the image features to be identified and the image features contained in the currently selected third feature cluster (i.e., the current sixth feature cluster).

[0406] The process of determining whether a specified object is contained in an image based on the sixth similarity scores of the image to be identified can be referred to in step S304 above, which describes the process of determining whether a specified object is contained in an image based on the second similarity scores of the image to be processed. This process will not be elaborated upon here.

[0407] In some embodiments, step S707 includes:

[0408] Determine the largest sixth similarity among the sixth similarities corresponding to the image to be identified, which is a third preset number; if all of the third preset number of largest sixth similarities are greater than the third preset similarity threshold, then determine that the image to be identified does not contain the specified object; otherwise, determine that the image to be identified contains the specified object.

[0409] In this embodiment of the application, the process of determining the third preset number of largest sixth similarities among the sixth similarities corresponding to the image to be identified can refer to the process of determining the first preset number of largest second similarities among the second similarities corresponding to the current image to be processed in step 1 above.

[0410] Furthermore, since the image features in the current sixth feature cluster are those that do not contain the specified object, if all of the third preset number of sixth similarities are greater than the third preset similarity threshold, it indicates that the image features of the image to be identified are similar to the image features of the image that does not contain the specified object. Therefore, it can be determined that the image to be identified does not contain the specified object.

[0411] If none of the third preset number of sixth similarities are greater than the third preset similarity threshold, it indicates that the image features of the image to be identified are not similar to the image features of an image that does not contain the specified object. Therefore, it can be determined that the image to be identified contains the specified object.

[0412] Thus, by using the third-to-last preset number of sixth similarity scores with relatively large values ​​to determine whether the image to be identified contains the specified object, the amount of computation can be reduced and the efficiency of image screening can be improved.

[0413] Correspondingly, when the image to be identified contains a specified object, the electronic device can obtain the recognition result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0414] In some embodiments, the method further includes:

[0415] If the image to be identified does not contain the specified object, add the image to the current third image set.

[0416] In this embodiment of the application, if it is determined that the image to be identified does not contain the specified object, the electronic device can add the image to be identified to the current third image set. In the subsequent process of performing the above modeling and filtering, each feature cluster can be determined according to the new current image set.

[0417] In this way, the image set can be dynamically updated. That is, the images that do not contain the specified object (i.e., negative samples) recorded in the image set can be continuously updated during the image screening process. In other words, the images that do not contain the specified object obtained in the image recognition process can be retained and participate in the subsequent screening process, thereby improving the accuracy of subsequent image recognition.

[0418] Referring to Figure 9, which is a schematic flowchart of an image detection process provided by an embodiment of this application, after obtaining a captured image (i.e., the image to be identified), feature extraction can be performed on the image to be identified to obtain its image features. Then, the electronic device can determine whether the image to be identified contains a specified object (i.e., perform feature filtering) based on the image features of the image to be identified and the image features in the feature set.

[0419] If the image to be identified contains a specified object, then the image features of the image to be identified can be matched with the image features of images in the image library (i.e., feature comparison) to obtain the recognition result of the image to be identified, and the detection process ends.

[0420] If the image to be identified does not contain the specified object, the image features of the image to be identified can be added to the feature set (i.e., add the feature set), and the detection process ends.

[0421] In the above detection process, the dashed line between the image library and feature comparison indicates that the images used in the feature comparison process are images pre-added to the image library based on the image screening method provided in this application. Correspondingly, during the image screening process, for each image added to the library (i.e., the image currently to be screened), the image can be input into a pre-trained quality assessment model to obtain the predicted sub-scores for that image. Based on the predicted sub-scores and the current weights of each quality assessment dimension, the total predicted score for the image is calculated (i.e., a score is given to the image).

[0422] Then, threshold filtering can be applied to the imported image. If the predicted total score of the imported image is less than the preset filtering threshold, the image is determined not to be retained, and the image filtering process ends.

[0423] If the predicted total score of the input image is not less than the preset filtering threshold, the input image is determined to be retained. Then, feature extraction can be performed on the input image to obtain its image features. The electronic device can determine whether the input image contains a specified object (i.e., perform feature filtering) based on the image features of the input image and the image features in the feature set.

[0424] If the image in the database contains the specified object, the image can be added to the image library (i.e., added to the image library), and the image filtering process ends.

[0425] If the image being imported does not contain the specified object, the image features of the imported image can be added to the feature set (i.e., add the feature set), and the image filtering process ends.

[0426] Based on the same inventive concept, this application provides an image filtering device. Referring to Figure 10, which is a structural diagram of an image filtering device provided in this application, the device includes:

[0427] The first image selection module 1001 is used to select an image that has not yet been filtered from all images to be filtered, and use it as the current image to be filtered.

[0428] The first scoring module 1002 is used to input the image to be screened into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the image to be screened based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein, the quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of the sample image; the total sample score of a sample image is obtained based on the initial values ​​of each sample sub-score and the weights of each quality assessment dimension of the sample image;

[0429] The filtering result determination module 1003 is used to determine the filtering result of the current image to be filtered as "retained" if the predicted total score of the current image to be filtered is within a preset range, and to adjust the current weights of each quality evaluation dimension based on the current filtering ratio and the target filtering ratio according to a reinforcement learning algorithm. If the predicted total score of the current image to be filtered is not less than a preset filtering threshold, the filtering result of the current image to be filtered is "not retained". If the predicted total score of the current image to be filtered is less than the preset filtering threshold, the filtering result of the current image to be filtered is "not retained". The preset range includes the preset filtering threshold. The current filtering ratio is the ratio of the number of images whose filtering result is "not retained" to the number of images whose filtering result has been determined. The module then returns to trigger the first image selection module 1001 until the filtering results of all images to be filtered are obtained.

[0430] The image adding module 1004 is used to obtain images to be added to the current image library based on the images selected for retention in the filtering results.

[0431] In some embodiments, the image adding module 1004 includes:

[0432] The first image selection submodule is used to select one currently unprocessed image from the images that are selected as the current image to be processed;

[0433] The first feature extraction submodule is used to extract features from the current image to be processed, and obtain the current first image features;

[0434] The first object determination submodule is used to determine whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features; wherein, the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; the images in the current first image set do not contain the specified object;

[0435] The first image addition submodule is used to determine that if the image to be processed contains the specified object, the image to be processed is an image to be added to the current image library, and to trigger the first image selection submodule.

[0436] In some embodiments, the first object determining submodule includes:

[0437] The first similarity calculation unit is used to calculate the first similarity between the current first image feature and the cluster centers of the current first feature clusters;

[0438] The first feature cluster selection unit is used to select the first feature cluster with the highest first similarity from the current first feature clusters;

[0439] The second similarity calculation unit is used to calculate the second similarity between the current first image feature and each image feature contained in the currently selected first feature cluster;

[0440] The first object determination unit is used to determine whether the current image to be processed contains a specified object based on the second similarity scores corresponding to the current image to be processed.

[0441] In some embodiments, the first object determining unit is specifically used for:

[0442] Determine the largest number of second similarities among the various second similarities corresponding to the current image to be processed;

[0443] If all of the first preset number of largest second similarities are greater than the first preset similarity threshold, then it is determined that the current image to be processed does not contain the specified object; otherwise, it is determined that the current image to be processed contains the specified object.

[0444] In some embodiments, the apparatus further includes:

[0445] The first image set update module is used to add the currently pending image to the current first image set if it is determined that the specified object is not contained in the currently pending image.

[0446] In some embodiments, the apparatus further includes:

[0447] The second feature extraction module is used to extract features from the current image to be screened before inputting the current image to be screened into the pre-trained quality assessment model and obtaining the predicted sub-scores output by each branch network in the quality assessment model, so as to obtain the current second image features.

[0448] The first object determination module is used to determine whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features; wherein, the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; the images in the current second image set do not contain the specified object;

[0449] The first scoring module 1002 is specifically used for:

[0450] If the specified object is contained in the current image to be screened, the current image to be screened is input into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model.

[0451] In some embodiments, the first object determining module includes:

[0452] The first similarity calculation submodule is used to calculate the third similarity between the current second image feature and the cluster centers of each of the current second feature clusters;

[0453] The first feature cluster selection submodule is used to select the corresponding second feature cluster with the highest third similarity from the current second feature clusters;

[0454] The second similarity calculation submodule is used to calculate the fourth similarity between the current second image feature and each image feature contained in the currently selected second feature cluster;

[0455] The second object determination submodule is used to determine whether the image to be filtered contains a specified object based on the fourth similarity scores corresponding to the image to be filtered.

[0456] In some embodiments, the second object determining submodule is specifically used for:

[0457] Determine the largest fourth similarity scores among the fourth similarity scores corresponding to the current image to be screened, which is a second preset number.

[0458] If all of the second preset number of largest fourth similarities are greater than the second preset similarity threshold, then it is determined that the current image to be filtered does not contain the specified object; otherwise, it is determined that the current image to be filtered contains the specified object.

[0459] In some embodiments, the apparatus further includes:

[0460] The second image set update module is used to add the currently selected image to the current second image set and trigger the first image selection module 1001 when it is determined that the currently selected image does not contain the specified object.

[0461] In some embodiments, the filtering result determination module 1003 is specifically used for:

[0462] Obtain the current state; where the current state includes: the current filtering ratio, the number of images that have been determined as the filtering results, and the current weight of each quality evaluation dimension;

[0463] Determine the expected benefit of taking each alternative action in the current state, as the current first expectation; wherein, an alternative action includes: the adjustment amount of the weights of each quality evaluation dimension;

[0464] Select the alternative action corresponding to the highest current first expectation as the action to be taken now;

[0465] Based on the actions required at present, the current weights of each quality evaluation dimension will be adjusted.

[0466] In some embodiments, the apparatus further includes:

[0467] The expectation update module is used to update the current first expectation based on the first formula after determining the action that needs to be taken.

[0468] The first formula is:

[0469] Q ′ (S,A)=Q(S,A)+η(R+θmaxQ(S′,A′)-Q(S,A));

[0470]

[0471] Q(S,A) represents the current first expectation, S represents the current state, and A represents the action to be taken; Q′(S,A) represents the updated first expectation obtained by updating the current first expectation; η represents the learning rate, θ represents the discount factor; R represents the reward; S′ represents the next state after obtaining the filtering result of the current image to be filtered, maxQ(S′,A′) represents the maximum value of the second expectation of the reward obtained when taking each alternative action in the next state S′, and A′ represents the alternative action corresponding to the maximum value of the second expectation;

[0472] st represents the constraint condition, R(S,A) means: the reward R is a function based on the current state S and the action A to be taken; f(z) represents the predicted total score of the current image to be screened, R c R represents the preset filtering threshold. min R represents the minimum value within the preset range. max R represents the maximum value within the preset range. min <Rc <R max M represents the target filtering ratio, p represents the current filtering ratio, and otherwise represents other cases.

[0473] In some embodiments, the quality assessment model is trained through the following steps:

[0474] Obtain a sample image containing a specified object; wherein the label of the sample image includes: the sample sub-score of the sample image on each quality evaluation dimension, and the sample total score of the sample image; the sample total score of a sample image is obtained based on the initial values ​​of the sample sub-scores and the weights of each quality evaluation dimension of the sample image.

[0475] For each sample image, the sample image is input into the quality assessment model of the initial structure to obtain the training sub-scores output by each branch network in the quality assessment model of the initial structure. Based on the initial values ​​of the weights of each quality assessment dimension represented by each branch network, the total training score of the sample image is calculated.

[0476] The loss value is calculated based on the training sub-scores and total training score of the sample image, as well as the sample sub-scores and total sample score of the sample image.

[0477] The model parameters of the initial structure quality evaluation model are adjusted based on the obtained loss value until convergence is achieved.

[0478] In some embodiments, the first scoring module 1002 is specifically used for:

[0479] Based on the current weights of each predicted sub-score and each quality evaluation dimension represented by each branch network, the total predicted score of the image to be screened is calculated according to the second formula.

[0480] The second formula is:

[0481] f(z)=w×(αf(x1)×…×βf(x i )×…×εf(x k ))+n;

[0482] f(z) represents the predicted total score of the image to be selected; k represents the number of each quality assessment dimension; f(x1) represents the predicted sub-score of the image to be selected in the first quality assessment dimension, and α represents the current weight of the first quality assessment dimension; f(x i f(x) represents the predicted sub-rating of the image to be filtered in the i-th quality assessment dimension, and β represents the current weight of the i-th quality assessment dimension; k) represents the predicted sub-score of the image to be filtered in the k-th quality evaluation dimension, ε represents the current weight of the k-th quality evaluation dimension; w is the preset first parameter, and n is the preset second parameter.

[0483] In some embodiments, each quality evaluation dimension includes at least one of the following: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image.

[0484] Based on the same inventive concept, this application provides an image recognition device. Referring to Figure 11, which is a structural diagram of an image recognition device provided in this application, the device includes:

[0485] Image acquisition module 1101 is used to acquire the image to be recognized;

[0486] The first feature extraction module 1102 is used to extract features from the image to be identified to obtain the features of the image to be identified;

[0487] The recognition result determination module 1103 is used to obtain the recognition result of the image to be recognized based on the matching result between the features of the image to be recognized and the image features of the images in the current image library; wherein the images in the current image library are obtained based on the image filtering method described in the first aspect above.

[0488] In some embodiments, the apparatus further includes:

[0489] The first similarity calculation module is used to calculate the fifth similarity between the features of the image to be identified and the cluster centers of each of the current third feature clusters before obtaining the recognition result of the image to be identified based on the matching result between the features of the image to be identified and the image features of the images in the current image library; wherein, each of the current third feature clusters is obtained by clustering the image features of the images contained in the current third image set; the images in the current third image set do not contain the specified object;

[0490] The first feature cluster selection module is used to select the third feature cluster with the highest similarity from the current third feature clusters;

[0491] The second similarity calculation module is used to calculate the sixth similarity between the features of the image to be identified and the image features contained in the currently selected third feature cluster;

[0492] The second object determination module is used to determine whether the image to be identified contains a specified object based on each sixth similarity corresponding to the image to be identified.

[0493] The recognition result determination module 1103 is specifically used for:

[0494] If the image to be identified contains a specified object, the identification result of the image to be identified is obtained based on the matching result between the features of the image to be identified and the image features of the images in the current image library.

[0495] In some embodiments, the second object determining module is specifically used for:

[0496] Determine the largest sixth similarity among the sixth similarities corresponding to the image to be identified, which is a third preset number.

[0497] If all of the third preset number of largest sixth similarities are greater than the third preset similarity threshold, then it is determined that the image to be identified does not contain the specified object; otherwise, it is determined that the image to be identified contains the specified object.

[0498] In some embodiments, the apparatus further includes:

[0499] The third image set update module is used to add the image to be identified to the current third image set if the image to be identified does not contain the specified object.

[0500] In the technical solution of this application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing the user's facial information are all carried out with the user's authorization.

[0501] It should be noted that the facial images in this embodiment are from a publicly available dataset.

[0502] This application also provides an electronic device, as shown in FIG12, including:

[0503] Memory 1201 is used to store computer programs;

[0504] When the processor 1202 executes the program stored in the memory 1201, it implements the steps of any of the above-described image filtering methods or image recognition methods.

[0505] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1202, the communication interface, and the memory 1201 communicating with each other via the communication bus.

[0506] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0507] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0508] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0509] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0510] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image filtering methods or image recognition methods.

[0511] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image filtering methods or image recognition methods described above.

[0512] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0513] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0514] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0515] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image filtering method, characterized in that, The method includes: selecting an image that has not yet been filtered from all images to be filtered, as the current image to be filtered; inputting the current image to be filtered into a pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculating the predicted total score of the current image to be filtered based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein, the quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of the sample image; the total sample score of a sample image is based on the initial values ​​of each sample sub-score and the weights of each quality assessment dimension of the sample image. If the predicted total score of the image to be filtered falls within a preset range, then the current state is obtained; where the current state includes: the current filtering ratio, the number of images whose filtering results have been determined, and the current weights of each quality evaluation dimension; the expected benefit of taking each alternative action in the current state is determined as the current first expectation; an alternative action includes: the adjustment amount of the weights of each quality evaluation dimension; the alternative action corresponding to the current largest first expectation is selected as the action to be taken; based on the action to be taken, the current weights of each quality evaluation dimension are adjusted; after determining the action to be taken, the current first expectation is updated based on the first formula; the first formula is: ； ； This indicates the primary expectation at present. Indicates the current state. Indicates the action that needs to be taken at this time; This represents the updated first expectation obtained by updating the current first expectation; Indicates the learning rate. Indicates the discount factor; Indicates a reward; This indicates the next state after obtaining the filtering results for the current image to be filtered. Indicate the next state The maximum of the second expected value of the payoff obtained when taking each alternative action. This represents the alternative action corresponding to the maximum value in the second expectation; Indicates constraints. Indicates: Reward Based on the current state And the actions that need to be taken now. The function; This represents the total predicted score for the image currently being filtered. This represents the preset filtering threshold. This represents the minimum value within the preset range. This represents the maximum value within the preset range, and ； Indicates the target filtering ratio. Indicates the current filtration ratio; Other cases are indicated; and if the predicted total score of the current image to be filtered is not less than the preset filtering threshold, the current image to be filtered is determined to be retained, and if the predicted total score of the current image to be filtered is less than the preset filtering threshold, the current image to be filtered is determined to be not retained; wherein, the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose filtering result is not retained to the number of images whose filtering result has been determined; return to the step of selecting an image that has not been filtered from all images to be filtered as the current image to be filtered, until the filtering results of all images to be filtered are obtained; based on the images whose filtering result is retained, the images to be added to the current image library are obtained.

2. The method according to claim 1, characterized in that, The step of obtaining images to be added to the current image library based on the images selected as retained by the filtering results includes: selecting an unprocessed image from the images selected as retained by the filtering results as the current image to be processed; extracting features from the current image to be processed to obtain current first image features; determining whether the current image to be processed contains a specified object based on the matching results of the current first feature clusters and the current first image features; wherein the current first feature clusters are obtained by clustering the image features of the images contained in the current first image set; the images in the current first image set do not contain the specified object; if the current image to be processed contains the specified object, then the current image to be processed is determined to be the image to be added to the current image library, and the step of selecting an unprocessed image from the images selected as retained by the filtering results as the current image to be processed is returned.

3. The method according to claim 2, characterized in that, The step of determining whether the specified object is contained in the current image to be processed based on the matching results of the current first feature clusters and the current first image features includes: calculating the first similarity between the current first image features and the cluster centers of the current first feature clusters; selecting the first feature cluster with the largest first similarity from the current first feature clusters; calculating the second similarity between the current first image features and the image features contained in the currently selected first feature clusters; and determining whether the specified object is contained in the current image to be processed based on the second similarity corresponding to the current image to be processed.

4. The method according to claim 3, characterized in that, The step of determining whether the current image to be processed contains a specified object based on each second similarity corresponding to the current image to be processed includes: determining the largest second similarity among each second similarity corresponding to the current image to be processed by a first preset number; if all of the first preset number of largest second similarities are greater than a first preset similarity threshold, then determining that the current image to be processed does not contain the specified object; otherwise, determining that the current image to be processed contains the specified object; and / or, the method further includes: if it is determined that the current image to be processed does not contain the specified object, adding the current image to be processed to the current first image set.

5. The method according to claim 1, characterized in that, Before inputting the current image to be screened into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model, the method further includes: extracting features from the current image to be screened to obtain current second image features; determining whether the current image to be screened contains a specified object based on the matching results of the current second feature clusters and the current second image features; wherein, the current second feature clusters are obtained by clustering the image features of the images contained in the current second image set; the images in the current second image set do not contain the specified object; the step of inputting the current image to be screened into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model includes: if the current image to be screened contains the specified object, inputting the current image to be screened into the pre-trained quality assessment model to obtain the predicted sub-scores output by each branch network in the quality assessment model.

6. The method according to claim 5, characterized in that, The step of determining whether the specified object is contained in the current image to be filtered based on the matching results of the current second feature clusters and the current second image features includes: calculating the third similarity between the current second image features and the cluster centers of the current second feature clusters; selecting the second feature cluster with the largest corresponding third similarity from the current second feature clusters; calculating the fourth similarity between the current second image features and the image features contained in the currently selected second feature clusters; and determining whether the specified object is contained in the current image to be filtered based on the fourth similarity corresponding to the current image to be filtered.

7. The method according to claim 6, characterized in that, The step of determining whether the specified object is contained in the current image to be filtered based on the fourth similarity scores corresponding to the current image to be filtered includes: determining the largest fourth similarity scores of a second preset number among the fourth similarity scores corresponding to the current image to be filtered; if all of the second preset number of largest fourth similarity scores are greater than a second preset similarity threshold, then it is determined that the specified object is not contained in the current image to be filtered; otherwise, it is determined that the specified object is contained in the current image to be filtered; and / or, the method further includes: if it is determined that the specified object is not contained in the current image to be filtered, adding the current image to be filtered to the current second image set, and returning to the step of selecting an image that has not yet been filtered from all images to be filtered as the current image to be filtered.

8. The method according to claim 1, characterized in that, The quality assessment model is trained through the following steps: acquiring sample images containing specified objects; wherein, the labels of the sample images include: sample sub-scores of the sample image on each quality assessment dimension, and the total sample score of the sample image; the total sample score of a sample image is obtained based on the initial values ​​of the sample sub-scores and the weights of each quality assessment dimension; for each sample image, the sample image is input into the initial structure of the quality assessment model to obtain the training sub-scores output by each branch network in the initial structure of the quality assessment model, and based on the initial values ​​of the training sub-scores and the weights of each quality assessment dimension represented by each branch network, ... Calculate the total training score of the sample image; calculate the loss value based on the training sub-scores and the total training score of the sample image, as well as the sample sub-scores and the total sample score of the sample image; adjust the model parameters of the initial structure quality assessment model based on the obtained loss value until convergence is achieved; and / or, the step of calculating the predicted total score of the current image to be screened based on the current weights of each predicted sub-score and each quality assessment dimension represented by each branch network includes: calculating the predicted total score of the current image to be screened according to the second formula based on the current weights of each predicted sub-score and each quality assessment dimension represented by each branch network; wherein, the second formula is: ； This represents the predicted total score for the image to be filtered; k represents the number of each quality assessment dimension. This represents the predicted sub-score of the image to be filtered in the first quality assessment dimension. This indicates the current weight of the first quality evaluation dimension; This represents the predicted sub-score of the image to be filtered on the i-th quality assessment dimension. This represents the current weight of the i-th quality evaluation dimension; This represents the predicted sub-score of the image to be filtered on the k-th quality assessment dimension. This represents the current weight of the k-th quality evaluation dimension; The first parameter is preset. The second parameter is a preset parameter; and / or, each quality evaluation dimension includes at least one of the following: image sharpness, image visibility, pitch angle of a specified object in the image, heading angle of a specified object in the image, and area proportion of a specified object in the image.

9. An image recognition method, characterized in that, The method includes: acquiring an image to be identified; extracting features from the image to be identified to obtain image features; and obtaining an identification result of the image to be identified based on the matching result between the image features of the image to be identified and the image features of images in the current image library; wherein the images in the current image library are obtained based on the image filtering method according to any one of claims 1-8.

10. The method according to claim 9, characterized in that, Before obtaining the recognition result of the image to be recognized based on the matching result between the features of the image to be recognized and the image features of the images in the current image library, the method further includes: calculating the fifth similarity between the features of the image to be recognized and the cluster centers of each of the current third feature clusters; wherein, each of the current third feature clusters is obtained by clustering the image features of the images contained in the current third image set; the images in the current third image set do not contain the specified object; selecting the third feature cluster with the largest fifth similarity from the current third feature clusters; calculating the sixth similarity between the features of the image to be recognized and the image features contained in the currently selected third feature cluster; determining whether the image to be recognized contains the specified object based on the sixth similarity corresponding to the image to be recognized; the method further includes: calculating the fifth similarity between the features of the image to be recognized and the image features of the images in the current image library; and calculating the fifth similarity between the features of the image to be recognized and the image features of the images in the current image library. The method for obtaining the recognition result of the image to be recognized based on the matching results between the features of the image to be recognized and the image features of the images in the current image library, includes: when the image to be recognized contains a specified object, obtaining the recognition result of the image to be recognized based on the matching results between the features of the image to be recognized and the image features of the images in the current image library; and / or, determining whether the image to be recognized contains a specified object based on each sixth similarity corresponding to the image to be recognized includes: determining the largest sixth similarity of a third preset number among the sixth similarities corresponding to the image to be recognized; if all of the largest sixth similarities of the third preset number are greater than a third preset similarity threshold, then determining that the image to be recognized does not contain a specified object; otherwise, determining that the image to be recognized contains the specified object; and / or, the method further includes: when the image to be recognized does not contain a specified object, adding the image to be recognized to the current third image set.

11. An image filtering device, characterized in that, The device includes: a first image selection module, configured to select an image that has not yet been selected from all images to be selected, as the current image to be selected; and a first scoring module, configured to input the current image to be selected into a pre-trained quality assessment model, obtain the predicted sub-scores output by each branch network in the quality assessment model, and calculate the predicted total score of the current image to be selected based on each predicted sub-score and the current weights of each quality assessment dimension represented by each branch network; wherein, the quality assessment model is trained based on the sample sub-scores of a sample image containing a specified object on each quality assessment dimension, and the total sample score of the sample image; the total sample score of a sample image is based on the sample sub-scores and the weights of each quality assessment dimension of the sample image. The initial value is obtained from the weight; the filtering result determination module is used to obtain the current state if the predicted total score of the image to be filtered is within a preset range; wherein, the current state includes: the current filtering ratio, the number of images whose filtering results have been determined, and the current weight of each quality evaluation dimension; determine the expected benefit obtained when taking each alternative action in the current state, as the current first expectation; an alternative action includes: the adjustment amount of the weight of each quality evaluation dimension; select the alternative action corresponding to the current largest first expectation as the action to be taken; adjust the current weight of each quality evaluation dimension based on the action to be taken; after determining the action to be taken, update the current first expectation based on the first formula; the first formula is: ； ； This indicates the primary expectation at present. Indicates the current state. Indicates the action that needs to be taken at this time; This represents the updated first expectation obtained by updating the current first expectation; Indicates the learning rate. Indicates the discount factor; Indicates a reward; This indicates the next state after obtaining the filtering results for the current image to be filtered. Indicate the next state The maximum of the second expected value of the payoff obtained when taking each alternative action. This represents the alternative action corresponding to the maximum value in the second expectation; Indicates constraints. Indicates: Reward Based on the current state And the actions that need to be taken now. The function; This represents the total predicted score for the image currently being filtered. This represents the preset filtering threshold. This represents the minimum value within the preset range. This represents the maximum value within the preset range, and ； Indicates the target filtering ratio. Indicates the current filtration ratio; Other cases are indicated; and if the predicted total score of the current image to be filtered is not less than the preset filtering threshold, the current image to be filtered is determined to be retained; and if the predicted total score of the current image to be filtered is less than the preset filtering threshold, the current image to be filtered is determined to be not retained; wherein, the preset range includes the preset filtering threshold; the current filtering ratio is: the ratio of the number of images whose filtering result is not retained to the number of images whose filtering result has been determined; return to trigger the first image selection module until the filtering results of all images to be filtered are obtained; the image adding module is used to obtain images to be added to the current image library based on the images whose filtering result is retained.

12. An image recognition device, characterized in that, The apparatus includes: a target image acquisition module for acquiring a target image; a first feature extraction module for extracting features from the target image to obtain target image features; and a recognition result determination module for obtaining a recognition result of the target image based on the matching result between the target image features and the image features of images in the current image library; wherein the images in the current image library are obtained based on the image filtering method according to any one of claims 1-8.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-8, or the method of any one of claims 9-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-8, or the method of any one of claims 9-10.

Images