Image classification method and device

Abstract

The embodiment of the invention provides an image classification method and device, and relates to the technical field of data processing, and the method comprises the steps: obtaining a first type of personnel characteristics of a to-be-classified personnel image; performing feature dimension reduction processing on the first type of personnel features of the to-be-classified personnel image to obtain a second type of personnel features of the to-be-classified personnel image; determining representative personnel images similar to the to-be-classified personnel images as candidate images based on the second type of personnel characteristics of the to-be-classified personnel images and the second type of personnel characteristics of representative personnel images in existing image types; and determining an image category to which the to-be-classified person image belongs based on the first category of person features of the to-be-classified person image and the first category of person features of the candidate image. By applying the scheme provided by the embodiment of the invention, the image classification efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image classification method and device. Background Art

[0002] In security and access control scenarios, image acquisition devices are generally deployed to continuously acquire images. In order to facilitate the management of images acquired by the image acquisition devices, the images can be classified. For example, images belonging to the same person can be classified into one image category. In this way, all images belonging to the same person can be queried in one image category, which brings convenience to subsequent applications.

[0003] In the related art, when classifying images captured by an image acquisition device, the image features of the persons in the image are generally compared with the image features of the persons in the images included in the existing image categories, and the image category to which the image belongs is determined based on the comparison result.

[0004] However, as time goes by, the number of images collected by image acquisition devices is increasing, and the number of images contained in existing image categories is also increasing. As a result, the scale of image features involved in comparison during image classification is getting larger and larger, and the amount of computation required for image classification is increasing, resulting in low efficiency of image classification. Summary of the invention

[0005] The purpose of the embodiment of the present application is to provide an image classification method and device to improve the efficiency of image classification. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides an image classification method, the method comprising:

[0007] Obtaining first-category person features of the person image to be classified;

[0008] Performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified;

[0009] Based on the second category of personnel features of the person image to be classified and the second category of personnel features of representative person images in existing image categories, determining a representative person image similar to the person image to be classified as a candidate image;

[0010] Based on the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the person image to be classified belongs is determined.

[0011] In a second aspect, an embodiment of the present application provides an image classification device, the device comprising:

[0012] A first-category personnel feature acquisition module, used to acquire first-category personnel features of the personnel image to be classified;

[0013] A second type of personnel feature acquisition module, used for performing feature dimension reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified;

[0014] A candidate image determination module, configured to determine, based on the second category of personnel features of the personnel image to be classified and the second category of personnel features of representative personnel images in existing image categories, a representative personnel image similar to the personnel image to be classified as a candidate image;

[0015] The image category determination module is used to determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0017] Memory, used to store computer programs;

[0018] The processor is used to implement the method described in the first aspect when executing the program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0021] It can be seen from the above that when the scheme provided in the embodiment of the present application is used for image classification, after obtaining the first category of personnel features of the person image to be classified, feature dimensionality reduction processing is performed on the obtained first category of personnel features to obtain the second category of personnel features of the person image to be classified. Then, based on the second category of personnel features of the person image to be classified and the second category of personnel features of the representative person images in the existing image categories, a representative person image similar to the person image to be classified is determined as a candidate image. Then, based on the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the person image to be classified belongs can be determined, thereby successfully achieving image classification.

[0022] Among them, on the first aspect, the first type of personnel features of the personnel images to be classified are subjected to feature dimensionality reduction processing to obtain the second type of personnel features with lower feature dimensions. Obviously, the computational complexity of processing the second type of personnel features with lower feature dimensions is lower than the computational complexity of processing the first type of personnel features with higher feature dimensions. Therefore, compared with directly determining the candidate images based on the comparison between the first type of personnel features, determining the candidate images based on the comparison between the second type of personnel features can effectively reduce the computational complexity required for feature comparison, thereby improving the efficiency of determining the candidate images; on the second aspect, when determining the candidate images, the personnel features involved in the comparison are the personnel features of a part of the representative personnel images in the existing image categories, rather than the personnel features of all the personnel images contained in the existing image categories. This can greatly reduce the number of personnel features involved in the comparison when determining the candidate images, thereby reducing the amount of data required to be processed during feature comparison, and improving the efficiency of determining the candidate images. Combining the above two aspects, the scheme provided by the embodiment of the present application can greatly improve the efficiency of determining the candidate images, thereby improving the efficiency of image classification based on the candidate images.

[0023] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0025] Figure 1 A schematic diagram of the process of the first image classification method provided in the embodiment of the present application;

[0026] Figure 2 A schematic diagram of a second image classification method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a neighbor graph provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a node search process provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of a third image classification method provided in an embodiment of the present application;

[0030] Figure 6 A schematic diagram of a fourth image classification method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0034] First, the execution subject of the solution provided in the embodiment of the present application is described.

[0035] The executor of the solution provided in the embodiment of the present application is: any electronic device with data processing, storage and other functions.

[0036] The image classification scheme provided in the embodiments of the present application is described in detail below.

[0037] See also Figure 1 , is a flow chart of the first image classification method provided in an embodiment of the present application, the method includes the following steps S101-S104.

[0038] Step S101: obtaining first-category person features of the person image to be classified.

[0039] First, the above-mentioned person images to be classified are explained.

[0040] From the perspective of image content, the person image to be classified can be a face image or a body image.

[0041] In terms of the image acquisition method, the image of the person to be classified can be an image selected from images captured by an image acquisition device, such as selecting an image containing a face or a body from the above images; it can also be a video frame selected from a video captured by an image acquisition device, such as capturing a video frame containing a face or a body from the above video.

[0042] In terms of the degree of image processing, the image of the person to be classified can be an original image without any processing, or, of course, can be an image obtained by intercepting the face area or body area in the original image.

[0043] The characteristics of the first type of personnel are explained below.

[0044] The first type of person features are image features extracted from the person images to be classified, and are different according to the images to be classified.

[0045] For example, if the person image to be classified is a face image, the first type of person features are face features; if the person image to be classified is a body image, the first type of person features are body features.

[0046] Specifically, an object detection algorithm may be used to detect a face region or a body region in an image, and then an object feature extraction algorithm may be used to extract features from the detected face region or body region to obtain first-category personnel features.

[0047] The above-mentioned object detection algorithm can be an algorithm for object detection based on the Fast R-CNN model, an algorithm for object detection based on the YoLo model, and the above-mentioned object feature extraction algorithm can be a convolutional neural network algorithm, a HOG (Histogram of Oriented Gradient, directional gradient histogram) algorithm, etc., which is not limited in the embodiment of the present application.

[0048] It should be noted that the first type of personnel features are image features used for image classification. The feature quality of the first type of personnel features has a greater impact on the accuracy of image classification. Therefore, in order to improve the accuracy of image classification, first type of personnel features with higher feature dimensions can be obtained. For example, 256-dimensional first type of personnel features can be obtained.

[0049] The first type of personnel features with higher dimensions obtained in this way have stronger personnel identification capabilities and higher feature quality, thereby improving the accuracy of image classification.

[0050] Step S102: performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified.

[0051] It can be seen that the second type of personnel characteristics are obtained after the first type of personnel characteristics are processed by feature dimensionality reduction. Therefore, the feature dimension of the second type of personnel characteristics is lower than that of the first type of personnel characteristics. Based on this, the first type of personnel characteristics can be called high-dimensional features, and the second type of personnel characteristics can be called low-dimensional features.

[0052] Specifically, the following methods can be used to perform feature dimensionality reduction processing on the first type of personnel features to obtain the second type of personnel features.

[0053] In one implementation, a linear dimensionality reduction algorithm such as PCA (principal component analysis) algorithm, LDA (Linear Discriminant Analysis) algorithm, FA (Factor Analysis) algorithm, etc. can be used to reduce the dimensionality of the first type of personnel characteristics to obtain the second type of personnel characteristics. A nonlinear dimensionality reduction algorithm such as KPCA (Kernelbased Principle Component Analysis) algorithm, KICA (kernel independent component analysis) algorithm, etc. can also be used to reduce the dimensionality of the first type of personnel characteristics to obtain the second type of personnel characteristics. Examples are not given one by one here.

[0054] In another implementation, the first type of personnel features of the person image to be classified may be input into a pre-trained feature quantization model to obtain the second type of personnel features of the person image to be classified output by the feature quantization model.

[0055] Among them, the training method of the above-mentioned feature quantization model is detailed in the subsequent embodiments and will not be described in detail here.

[0056] In this way, by inputting the first type of personnel features of the image of the person to be classified into the pre-trained feature quantization model, the second type of personnel features of the image of the person to be classified output by the feature quantization model can be easily obtained, thereby improving the efficiency of obtaining the second type of personnel features.

[0057] Step S103: Based on the second category personnel features of the person image to be classified and the second category personnel features of representative person images in existing image categories, a representative person image similar to the person image to be classified is determined as a candidate image.

[0058] The above-mentioned existing image categories are pre-divided image categories, and each image category includes a number of person images belonging to the same person.

[0059] The representative person images described above are described below.

[0060] The representative person image is a representative image in an image category. Each image category may include one representative person image or multiple representative person images whose number is not greater than a preset number threshold. The method for determining the preset number threshold is described in detail in the following Figure 5 This is introduced in step S508 in the illustrated embodiment and will not be described in detail here.

[0061] In one case, for each image category, the image quality representation values ​​of all the person images contained in the image category can be obtained, and the first preset number threshold of person images with the largest corresponding image quality representation values ​​are determined as representative person images of the image category.

[0062] The embodiment of the present application does not limit the method for determining the image quality characterization value of the person image, and is explained below by way of example.

[0063] The first method can determine the blur degree, lighting range, face integrity, face size, body posture and other information of the person image, perform weighted calculation on the obtained information according to the preset weights corresponding to the obtained information, and use the calculation result as the image quality representation value of the person image.

[0064] In the second method, an image quality evaluation algorithm such as an SSIM (structural similarity index) algorithm or an FSIM (feature similarity index merge) algorithm may be used to determine an image quality characterization value of the person image.

[0065] In a third method, the person image may be input into a pre-trained image quality assessment model to obtain an image quality representation value of the person image output by the image quality assessment model.

[0066] Among them, the above-mentioned model can be a model obtained by training a preset neural network model with sample person images as input and the image quality annotation values ​​of the sample person images, and is used to determine the image quality characterization values ​​of the person images.

[0067] In one case, during the training process, a higher image quality annotation value can be set for high-quality sample person images with characteristics such as high clarity, low facial tilt, and less facial area occlusion. In this way, the model can learn the characteristics of high-quality person images during the training process, and the trained model can output higher image quality characterization values ​​for person images with high clarity, low facial tilt, and less facial area occlusion.

[0068] The method for determining the candidate images is described below.

[0069] In one implementation, the similarity between the second category personnel features of the person image to be classified and the second category personnel features of representative person images in existing image categories can be calculated, and the representative person images corresponding to the largest preset number of similarities are determined as candidate images.

[0070] The similarity may be cosine similarity, or may be similarity determined based on Euclidean distance, Manhattan distance, etc. between features, which is not limited in the embodiments of the present application.

[0071] The above preset quantity can be set by the staff based on actual needs.

[0072] Specifically, the larger the above preset number is, the smaller the number of candidate images determined is, and the smaller the data processing amount of subsequent image classification is, so that the efficiency of image classification is higher and the accuracy of image classification is lower; conversely, the smaller the above preset number is, the more the number of candidate images determined is, and the larger the data processing amount of subsequent image classification is, so that the efficiency of image classification is lower and the accuracy of image classification is higher.

[0073] Therefore, the preset similarity may be determined based on the actual requirements of the current scene for image classification efficiency and the actual requirements for image classification accuracy.

[0074] For example, if the current scene has a higher actual requirement for image classification efficiency and a lower actual requirement for image classification accuracy, the above preset number can be set to a higher value; if the current scene has a lower actual requirement for image classification efficiency and a higher actual requirement for image classification accuracy, the above preset number can be set to a lower value.

[0075] In another implementation, the similarity between the second category personnel features of the person image to be classified and the second category personnel features of the representative person images in the existing image categories may be calculated, and the representative person images whose corresponding similarities are greater than a preset similarity may be determined as candidate images.

[0076] Similar to the previous implementation, the preset similarity can be determined based on the actual requirements of the current scene for image classification efficiency and the actual requirements for image classification accuracy, which will not be repeated here.

[0077] In another embodiment, based on the graph structure of the constructed neighbor graph, similar features of the second type of personnel features of the personnel image to be classified can be searched in the neighbor graph, and the representative personnel image corresponding to the searched similar features is used as the candidate image. Figure 2 Steps S203 to S204 in the illustrated embodiment are not described in detail here.

[0078] Step S104: determining the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0079] The first type of personnel features of each representative personnel image may be pre-stored, so after a candidate image is determined from the representative personnel images, the first type of personnel features of the determined candidate image may be directly read.

[0080] Specifically, multiple methods may be used to determine the image category to which the person image to be classified belongs.

[0081] In one implementation, the similarity between the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image can be calculated, and it is determined whether the maximum value of the obtained similarities is greater than a preset similarity threshold. If so, it is determined that the image category to which the person image to be classified belongs is the image category to which the target image belongs; otherwise, it is determined that the person image to be classified belongs to a new image category, and the person image to be classified is determined as a representative person image of the new image category.

[0082] The target image is a candidate image to which the first type of personnel feature corresponding to the maximum value belongs. The similarity threshold can be set by the staff according to the actual needs of the current scene.

[0083] It should be noted that if it is determined that the person image to be classified belongs to a new image category, the new image category only contains one person image. Therefore, the person image to be classified can be directly determined as the representative person image of the new image category.

[0084] It can be seen that first, the target image with the greatest similarity between the corresponding first-category personnel feature and the first-category personnel feature of the person image to be classified is determined from the candidate images, and then, in combination with the preset similarity threshold, when the above similarity is greater than the similarity threshold, the image category to which the person image to be classified belongs is determined as the image category to which the target image belongs. In this way, on the one hand, it can be ensured that the image category to which the person image to be classified belongs can be determined as the image category to which the target image with the greatest similarity between the corresponding first-category personnel feature and the first-category personnel feature of the person image to be classified belongs, which is conducive to improving the accuracy of image classification; on the other hand, it is further ensured based on the preset similarity threshold that the above similarity needs to meet certain requirements, further improving the accuracy of image classification.

[0085] In another embodiment, after obtaining the similarity between the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the target image having the greatest similarity between the corresponding first category of personnel features and the first category of personnel features of the person image to be classified belongs can be directly determined as the image category to which the person image to be classified belongs.

[0086] In another implementation manner, if the number of determined candidate images is 1, the image category to which the person image to be classified belongs may be directly determined to be the image category to which the determined candidate image belongs.

[0087] It can be seen from the above that when the scheme provided in the embodiment of the present application is used for image classification, after obtaining the first category of personnel features of the person image to be classified, feature dimensionality reduction processing is performed on the obtained first category of personnel features to obtain the second category of personnel features of the person image to be classified. Then, based on the second category of personnel features of the person image to be classified and the second category of personnel features of the representative person images in the existing image categories, a representative person image similar to the person image to be classified is determined as a candidate image. Then, based on the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the person image to be classified belongs can be determined, thereby successfully achieving image classification.

[0088] Among them, on the first aspect, the first type of personnel features of the personnel images to be classified are subjected to feature dimensionality reduction processing to obtain the second type of personnel features with lower feature dimensions. Obviously, the computational complexity of processing the second type of personnel features with lower feature dimensions is lower than the computational complexity of processing the first type of personnel features with higher feature dimensions. Therefore, compared with directly determining the candidate images based on the comparison between the first type of personnel features, determining the candidate images based on the comparison between the second type of personnel features can effectively reduce the computational complexity required for feature comparison, thereby improving the efficiency of determining the candidate images; on the second aspect, when determining the candidate images, the personnel features involved in the comparison are the personnel features of a part of the representative personnel images in the existing image categories, rather than the personnel features of all the personnel images contained in the existing image categories. This can greatly reduce the number of personnel features involved in the comparison when determining the candidate images, thereby reducing the amount of data required to be processed during feature comparison, and improving the efficiency of determining the candidate images. Combining the above two aspects, the scheme provided by the embodiment of the present application can greatly improve the efficiency of determining the candidate images, thereby improving the efficiency of image classification based on the candidate images.

[0089] In addition, after determining the candidate image, the solution provided in the embodiment of the present application ultimately determines the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image, that is, the features ultimately used to determine the image category to which the person features to be classified belong are still the first category of person features. Since the first category of person features have a higher feature dimension than the second category of person features and have a stronger ability to identify people, the final image category judgment based on the first category of person features can improve the accuracy of image classification.

[0090] In summary, the overall process of this solution is: based on the comparison between the second-category personnel features with lower feature dimensions, candidate images are determined from representative personnel images in existing image classifications, and then based on the comparison between the first-category personnel features with higher feature dimensions, it is determined whether the image category to which the image of the person to be classified belongs is the image category to which the candidate image belongs. In this way, based on the above two feature comparisons, both the efficiency and accuracy of image classification are taken into account, and both the efficiency and accuracy of image classification are significantly improved.

[0091] The following steps A to F are used to introduce the training method of the feature quantization model mentioned in the above step S102.

[0092] Step A: Obtain the first category of personnel features of the sample personnel images and the classification annotations of the sample personnel images.

[0093] The classification annotation of the sample person image is the true value of the image category to which the sample person image belongs.

[0094] Step B: Input the first type of personnel features of the sample personnel image into a preset neural network model for quantization processing to obtain the second type of personnel features of the sample personnel image output by the neural network model.

[0095] Step C: Input the second type of personnel features of the sample personnel image into a pre-trained image classification model for classification, and obtain the classification result of the sample personnel image output by the image classification model.

[0096] In this step, the classification result of the sample person image output by the image classification model is, that is, the prediction value of the image classification model for the image category to which the sample person image belongs.

[0097] Step D: Obtain a first loss value based on the similarity between the first category of personnel features and the similarity between the second category of personnel features of the sample personnel images.

[0098] Specifically, the first similarity between the first type of personnel features of the sample personnel images and the second similarity between the second type of personnel features of the sample personnel images can be calculated, and then the difference between the first similarity and the second similarity between the sample personnel images can be determined to obtain the first loss value.

[0099] For example, the first loss value may be a total value, an average value, etc. of the difference between the first similarity and the second similarity between the sample person images.

[0100] Step E: Based on the classification annotation and classification results of the sample person image, a second loss value is obtained.

[0101] Since the classification label of the sample person image is the true value of the image category to which the sample person image belongs, and the classification result of the sample person image is the predicted value of the image category to which the sample person image belongs, the second loss value can be obtained based on the difference between the classification label of the sample person image and the classification result.

[0102] Step F: Adjusting model parameters of the neural network model based on the first loss value and the second loss value.

[0103] Specifically, after adjusting the model parameters of the above neural network model, step B to step F can be executed repeatedly until the preset training end condition is met to obtain a feature quantization model.

[0104] When the training ends, the number of training times reaches the preset number, the first loss value and the second loss value meet the preset requirements, etc. The preset requirements may be that the first loss value and the second loss value are both less than their corresponding preset thresholds, or the sum of the first loss value and the second loss value is less than the preset threshold, etc.

[0105] It can be seen that, on the first hand, the first loss value is determined based on the similarity between the first category of personnel features and the second category of personnel features of the sample personnel image. Training the neural network model based on the first loss value can enable the model to learn the association between the similarity between the second category of personnel features after quantization of the first category of personnel features of the sample personnel image and the similarity between the original first category of personnel features, thereby ensuring that the similarity between the second category of personnel features after quantization is aligned with the similarity between the first category of personnel features, so that the second category of personnel features after quantization have sufficient distinctiveness; on the second hand, the second loss value is determined based on the classification annotation and classification results of the sample personnel image. Training the neural network model based on the second loss value can enable the model to learn the association between the second low-dimensional features of the sample image and the image category to which the sample personnel image belongs, thereby ensuring that the second category of personnel features after quantization can effectively distinguish personnel images belonging to different image categories in its feature space.

[0106] In summary, after the feature quantization model is obtained through model training based on the first loss value and the second loss value, the second type of personnel features with high discrimination of the image category of the personnel image and aligned with the similarity of the corresponding first type of personnel features can be obtained based on the feature quantization model. That is, more reasonable and accurate second type of personnel features can be obtained, thereby improving the accuracy of subsequent image classification based on the second type of personnel features.

[0107] exist Figure 1On the basis of the embodiment shown, when determining the candidate image based on the second type of personnel features of the image of the person to be classified and the second type of personnel features of the representative person images in the existing image categories, similar features of the second type of personnel features of the image of the person to be classified can be searched in the neighbor graph based on the graph structure of the constructed neighbor graph, and the representative person images corresponding to the similar features searched are used as candidate images. In view of the above situation, the embodiment of the present application provides a second image classification method.

[0108] See also Figure 2 , is a flow chart of a second image classification method provided in an embodiment of the present application, the method includes the following steps S201-S205.

[0109] Step S201: obtaining first-category person features of the person image to be classified.

[0110] Step S202: performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified.

[0111] The above steps S201-S202 are the same as the above steps Figure 1 In the illustrated embodiment, steps S101 and S102 are the same and will not be described in detail here.

[0112] Step S203: According to the graph structure of the constructed neighbor graph, similar features of the second type of personnel features of the person image to be classified are searched among the second type of personnel features included in the attributes of the nodes in the neighbor graph.

[0113] Among them, the neighbor graph is composed of nodes and the relationships between nodes, which are explained below.

[0114] Each node corresponds to a representative person image in the existing image category, and the attributes of the node may include: the second type of person features of the corresponding representative person image.

[0115] In one case, the attributes of the node may also include: first-category personnel features of the corresponding representative personnel image.

[0116] The relationship between nodes can also be called the edge between nodes, which is established based on the similarity between the second type of personnel features of the representative personnel images corresponding to the nodes. The comparison in the embodiment of the present application is not limited and is introduced below by way of example.

[0117] In one case, a relationship may be established between nodes whose similarity between the second type of personnel features of the corresponding representative personnel images is greater than a preset threshold.

[0118] In another case, for each node, a preset number of nodes having the highest similarity between the second type of personnel features of the corresponding representative personnel image and the second type of personnel features of the representative personnel image corresponding to the node can be determined, and a relationship can be established between the determined node and the node.

[0119] A neighbor graph can have a variety of graph structures. The following introduces two existing neighbor graphs with different graph structures by giving examples.

[0120] In one case, the existing neighbor graph may be a K-nearest neighbor graph, in which each node has a relationship with K nodes with the highest similarity to the node.

[0121] The similarity between the above nodes refers to the similarity between the second type of personnel features of the representative personnel images corresponding to the nodes.

[0122] In another case, the existing neighbor graph may be HNSW (Hierarchical Navigable Small Worldgraphs). The graph structure of HNSW is a hierarchical structure. Each layer of subgraph contains a number of nodes and the relationships between nodes. The number of nodes contained in each layer of subgraph increases from top to bottom. The next layer of subgraph contains all the nodes of the previous layer of subgraph, and the bottom layer of subgraph contains all the nodes.

[0123] Combine the following Figure 3 , a more intuitive explanation of the graph structure of the above HNSW is given.

[0124] See also Figure 3 , which is a schematic diagram of a neighbor graph provided in an embodiment of the present application.

[0125] Figure 3 The black origin in the middle represents the node in the neighbor graph. It can be seen that Figure 3 There are three layers of subgraphs in total, namely L0, L1 and L2. The top subgraph L2 contains only 2 nodes. The next layer of subgraph L1 of L2 contains 4 nodes including the nodes in L2. The bottom subgraph L0 contains 8 nodes including the nodes in L1 and L2. The above 8 nodes are all nodes.

[0126] Specifically, depending on the different graph structures of the constructed neighbor graph, the method of searching for similar features of the second type of personnel features of the person image to be classified in the second type of personnel features included in the attributes of the nodes in the neighbor graph is also different, which is explained in detail below.

[0127] In one implementation, if the existing neighbor graph is a K-nearest neighbor graph, each node in the K-nearest neighbor graph can be traversed to determine the first node having the highest similarity between the second-category personnel features contained in the attributes and the second-category personnel features of the image to be classified, and then determine the second node having a relationship with the node, and the second-category personnel features contained in the attributes of the determined first node and the second node are similar features of the second-category personnel features of the image to be classified.

[0128] In another embodiment, if the constructed neighbor graph is HNSW, any node in the topmost subgraph of the neighbor graph can be used as the starting node, and the most similar node is searched among the nodes in the topmost subgraph that have a relationship with the starting node. Then, the searched node is used as the starting node of the next subgraph, and the most similar node is searched among the nodes in the next subgraph that have a node relationship with the starting node. The steps of using the searched node as the starting node of the next subgraph and searching for the most similar node among the nodes in the next subgraph that have a node relationship with the starting node are repeated until the most similar node is searched in the bottom subgraph, and the second type of personnel features contained in the attributes of the most similar node in the bottom subgraph and the node that has a relationship with the most similar node are determined as similar features of the second type of personnel features of the personnel image to be classified.

[0129] The most similar node is the node with the highest similarity between the second type of personnel features included in the attribute and the second type of personnel features of the image to be classified.

[0130] Combine the following Figure 4 , which provides a more intuitive explanation of the above node search process.

[0131] See also Figure 4 , which is a schematic diagram of a node search process provided in an embodiment of the present application.

[0132] Figure 4 The nodes represented by the white dots in each layer of the subgraph are the most similar nodes searched in the subgraph of that layer.

[0133] The node search process of each layer of subgraph is explained below.

[0134] For L2:

[0135] L2 is the top-level subgraph. First, select any starting node 1 in L2, and search for the most similar node (the node represented by the white dot in L2) among the nodes that have a node relationship with the starting node 1.

[0136] For L1:

[0137] L1 contains all nodes of L2, the most similar node in L2 is determined as the starting node 2 in L1, and then the most similar node is searched among the nodes in L1 that have a node relationship with the starting node 2.

[0138] For L0:

[0139] Similarly, L0 includes all nodes of L1, the most similar node in L1 is determined as the starting node 3 in L0, and then the most similar node is searched among the nodes in L0 that have a node relationship with the starting node 3.

[0140] At this time, since L0 is determined to be the lowest level subgraph, all nodes that have a relationship with the most similar node in L0 can be determined, and then the most similar nodes in L0 and the nodes that have a relationship with the most similar nodes ( Figure 4 The second type of personnel features contained in the attributes of all nodes in the dotted box in L0 are determined as similar features of the second type of personnel features of the person image to be classified.

[0141] Step S204: the representative person image corresponding to the node to which the similar feature is searched is determined as a representative person image similar to the person image to be classified as a candidate image.

[0142] Step S205: Determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0143] The above steps S204-S205 are the same as the above steps Figure 1 In the illustrated embodiment, steps S103 and S104 are the same and will not be described in detail here.

[0144] As can be seen from the above, when the solution provided by the embodiment of the present application is used to determine the candidate image, the similar features of the second type of personnel features of the image of the person to be classified can be searched in the neighbor graph according to the graph structure of the constructed neighbor graph, and the representative person image corresponding to the node to which the similar features are searched is used as the candidate image. Since the neighbor graph is pre-constructed, the similar features of the second type of personnel features of the image of the person to be classified can be quickly determined based on the relationship between the nodes in the neighbor graph, thereby utilizing the high query rate of the neighbor graph to improve the efficiency of determining the candidate image.

[0145] In addition, on the one hand, the nodes in the neighbor graph correspond to a part of the representative personnel images in the existing image categories, rather than all the personnel images included in the existing image categories, which reduces the number of nodes in the neighbor graph and reduces the storage resource usage of the neighbor graph; on the other hand, since the second type of personnel features are features with lower feature dimensions obtained by compressing the first type of personnel features, the attributes of the nodes in the neighbor graph are set based on the second type of personnel features representing the personnel images, which further reduces the storage resource usage of the neighbor graph.

[0146] exist Figure 2 On the basis of the embodiment shown, after determining the image category to which the person image to be classified belongs, the neighbor graph can also be updated when the person image to be classified belongs to an existing first image category and it is determined that the person image to be classified needs to be set as a representative person image of the first image category. In view of the above situation, the embodiment of the present application provides a third image classification method.

[0147] See also Figure 5 , is a flow chart of a third image classification method provided in an embodiment of the present application, the method includes the following steps S501-S509.

[0148] Step S501: obtaining first-category person features of the person image to be classified.

[0149] Step S502: performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified.

[0150] Step S503: According to the graph structure of the constructed neighbor graph, similar features of the second type of personnel features of the person image to be classified are searched in the second type of personnel features included in the attributes of the nodes in the neighbor graph.

[0151] Step S504: The representative person image corresponding to the node to which the similar feature is searched is determined as a representative person image similar to the person image to be classified as a candidate image.

[0152] Step S505: Determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0153] The above steps S501 to S505 are the same as the above Figure 2 In the illustrated embodiment, steps S201 to S205 are the same and will not be described in detail here.

[0154] Step S506: Determine whether the person image to be classified belongs to the existing first image category. If yes, execute step S507.

[0155] Step S507: Determine whether it is necessary to set the person image to be classified as a representative person image of the first image category. If yes, execute step S508.

[0156] Specifically, the target image quality characterization value of the image of the person to be classified can be obtained, and it can be determined whether the target image quality characterization value is greater than the minimum image quality characterization value of the representative person image of the first image category. If it is greater, it is determined that the image of the person to be classified needs to be set as the representative person image of the first image category.

[0157] The meaning and method of obtaining the above image quality characterization values ​​have been described in the previous section. Figure 1 This is described in step S103 in the illustrated embodiment and will not be described again here.

[0158] It can be seen that when the target image quality representation value of the person image to be classified is greater than the minimum image quality representation value of the representative person image of the first image category, it can be considered that the image quality of the person image to be classified is high, that is, the representative ability for the image category is strong. Therefore, it can be determined at this time that the person image to be classified needs to be set as the representative person image of the first image category. In this way, the person image to be classified can be set as the representative person image of the first image category in a timely manner when the image quality of the person image to be classified is high, which is conducive to improving the accuracy of subsequent image classification based on the representative person image.

[0159] Step S508: adding the person images to be classified to the representative person images of the first image category.

[0160] Specifically, the following method may be used to add the person image to be classified to the representative person image of the first image category.

[0161] In one implementation, it can be determined whether the number of representative person images of the first image category is less than a preset number threshold. If not, the representative person image corresponding to the lowest image quality characterization value is replaced with the person image to be classified. Otherwise, the person image to be classified is directly added to the representative person images of the first image category.

[0162] For example, if the above-mentioned preset quantity threshold is 3, and the number of representative personnel images of the first image category is 3, then the number of representative personnel images of the first image category is not less than the preset quantity threshold, and the representative personnel image p1 corresponding to the lowest image quality characterization value in the first image category can be replaced with the personnel image p2 to be classified; for another example, if the number of representative personnel images of the first image category is 2, then the number of representative personnel images of the first image category is less than the preset quantity threshold, and the personnel image p2 to be classified can be directly added to the representative personnel images of the first image category.

[0163] It can be seen that the above method can make the number of representative person images of the image category approach the preset number threshold and not exceed the preset number threshold.

[0164] In one case, the preset quantity threshold may be determined based on the total number of images included in all existing image categories and is inversely proportional to the total number of images.

[0165] That is, the greater the total number of images included in all existing image categories, the smaller the preset number threshold can be; and the smaller the total number of images included in all existing image categories, the larger the preset number threshold can be.

[0166] For example, when the total number of images is less than one million, the preset number threshold may be determined as 3; and when the total number of images is greater than one hundred million, the preset number threshold may be determined as 1.

[0167] On the one hand, the larger the total number of images mentioned above is, the more image categories there are. Since the representative images of the existing image categories correspond one-to-one to the nodes in the neighbor graph, a smaller preset number threshold is determined in this way, that is, the number of representative images of the existing image categories is reduced, thereby reducing the number of nodes in the neighbor graph, preventing the search efficiency from being too large due to the neighbor graph being too large, which is beneficial to improving the image classification efficiency; conversely, the smaller the total number of images mentioned above is, the fewer the existing image categories are. At this time, a larger preset number threshold is determined, that is, the number of representative images of the existing image categories is increased, thereby increasing the number of nodes in the neighbor graph, preventing the search accuracy from being reduced due to the neighbor graph being too small, which is beneficial to improving the accuracy of image classification.

[0168] In summary, determining a preset number threshold that is inversely proportional to the total number of images can take into account both the efficiency and accuracy of image classification, thereby improving the rationality and practicality of the solution.

[0169] In another implementation, the person images to be classified may be directly added to the representative person images of the first image category.

[0170] It can be seen that, in this way, it can be determined to replace the representative person images of the first image category with the person images to be classified or to add the person images to be classified to the representative person images according to the number of representative person images of the first image category. On the one hand, when the number of representative person images is less than the preset number threshold, the person images to be classified can be directly added to the representative person images to ensure that the number of representative person images included in the image category is maintained at the preset number threshold, thereby avoiding too few representative person images included in the image category; on the other hand, when the number of representative person images is not less than the preset number threshold, the representative person images can be replaced with the person images to be classified, so that the existing representative person images can be continuously replaced with images with higher image quality, so that the representative person images are always updated with images with higher image quality, further improving the accuracy of subsequent image classification based on the representative person images.

[0171] Step S509: Update the neighbor graph based on the target node corresponding to the person image to be classified.

[0172] It can be seen from the above step S508 that there are two cases when adding the person image to be classified to the representative person image of the first image category. The updating of the neighbor graph in these two cases is respectively described below.

[0173] In one case, if the representative person image corresponding to the lowest image quality characterization value in the first image category is replaced with the person image to be classified, the node corresponding to the replaced representative person image in the neighbor graph can be replaced with the target node corresponding to the person image to be classified, the relationship between the node corresponding to the replaced representative person image and other nodes is deleted, and the attributes of the target node are set to include the second type of personnel features of the person image to be classified, and based on the similarity between the second type of personnel features included in the attributes of the target node and the second type of personnel features included in the attributes of the nodes in the neighbor graph, the relationship between the target node and the nodes in the neighbor graph is established.

[0174] In another case, if the image of the person to be classified is directly added to the representative person image of the first image category, a target node corresponding to the image of the person to be classified can be created, and the attributes of the target node are set to include the second-category personnel features of the image of the person to be classified, and based on the similarity between the second-category personnel features included in the attributes of the target node and the second-category personnel features included in the attributes of the nodes in the neighbor graph, the relationship between the target node and the nodes in the neighbor graph is established.

[0175] It can be seen from the above that when the scheme provided by the embodiment of the present application is used for image classification, when the image of the person to be classified belongs to the existing first image category and it is determined that the image of the person to be classified needs to be set as the representative person image of the first image category, the neighbor graph can be updated in time based on the target node corresponding to the image of the person to be classified. That is, as the image classification scheme proceeds, the neighbor graph can be continuously updated and optimized, further improving the accuracy of subsequent determination of candidate images based on the neighbor graph.

[0176] exist Figure 2 On the basis of the embodiment shown, after determining the image category to which the person image to be classified belongs, the neighbor graph can also be updated when the person image to be classified belongs to a new second image category. In view of the above situation, the embodiment of the present application provides a fourth image classification method.

[0177] See also Figure 6 , is a flow chart of the fourth image classification method provided in an embodiment of the present application, and the above method includes the following steps S601-S608.

[0178] Step S601: obtaining first-category person features of the person image to be classified.

[0179] Step S602: performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified.

[0180] Step S603: According to the graph structure of the constructed neighbor graph, similar features of the second type of personnel features of the person image to be classified are searched in the second type of personnel features included in the attributes of the nodes in the neighbor graph.

[0181] Step S604: the representative person image corresponding to the node to which the similar feature is searched is determined as a representative person image similar to the person image to be classified as a candidate image.

[0182] Step S605: Determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0183] The above steps S601 to S605 are the same as the above Figure 2 In the illustrated embodiment, steps S201 to S205 are the same and will not be described in detail here.

[0184] Step S606: Determine whether the person image to be classified belongs to the new second image category. If yes, execute step S607.

[0185] Since the nodes in the existing neighbor graph correspond one-to-one to the representative person images in the existing image categories, when it is determined that the person image to be classified belongs to the new second image category, the person image to be classified can be directly determined as the representative person image of the new second image category, and a new node can be created in the existing neighbor graph.

[0186] Step S607: creating a target node corresponding to the person image to be classified, and setting the attributes of the target node to include the second type of person features of the person image to be classified.

[0187] Step S608: Establishing a relationship between the target node and the nodes in the neighbor graph based on the similarity between the second type of personnel features included in the attributes of the target node and the second type of personnel features included in the attributes of the nodes in the neighbor graph.

[0188] It can be seen that through steps S607 and S608, new nodes and new relationships are established in the existing neighbor graph, thereby realizing incremental update of the existing neighbor graph.

[0189] It can be seen from the above that when the scheme provided by the embodiment of the present application is used for image classification, when the image of the person to be classified belongs to the new second image category, a target node corresponding to the image of the person to be classified can be created, and the relationship between the target node and the nodes in the neighbor graph can be established, so that the existing neighbor graph can be incrementally updated in a timely manner. That is, as the image classification scheme proceeds, the existing neighbor graph can be continuously incrementally updated, thereby improving the accuracy of subsequent determination of candidate images based on the neighbor graph.

[0190] Corresponding to the above-mentioned image classification method, an embodiment of the present application also provides an image classification device.

[0191] See also Figure 7 , is a schematic diagram of the structure of an image classification device provided in an embodiment of the present application, the device includes the following modules:

[0192] A first-category personnel feature acquisition module 701, used to acquire first-category personnel features of the personnel image to be classified;

[0193] The second type of personnel feature acquisition module 702 is used to perform feature dimension reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified;

[0194] A candidate image determination module 703 is used to determine a representative person image similar to the person image to be classified as a candidate image based on the second type of person features of the person image to be classified and the second type of person features of representative person images in existing image categories;

[0195] The image category determination module 704 is used to determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

[0196] It can be seen from the above that when the scheme provided in the embodiment of the present application is used for image classification, after obtaining the first category of personnel features of the person image to be classified, feature dimensionality reduction processing is performed on the obtained first category of personnel features to obtain the second category of personnel features of the person image to be classified. Then, based on the second category of personnel features of the person image to be classified and the second category of personnel features of the representative person images in the existing image categories, a representative person image similar to the person image to be classified is determined as a candidate image. Then, based on the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the person image to be classified belongs can be determined, thereby successfully achieving image classification.

[0197] Among them, in the first aspect, the first type of personnel features of the personnel images to be classified are subjected to feature dimensionality reduction processing to obtain the second type of personnel features with lower feature dimensions. Obviously, the computational complexity of processing the second type of personnel features with lower feature dimensions is lower than the computational complexity of processing the first type of personnel features with higher feature dimensions. Therefore, compared with directly determining the candidate images based on the comparison between the first type of personnel features, determining the candidate images based on the comparison between the second type of personnel features can effectively reduce the computational complexity required for feature comparison and improve the efficiency of determining the candidate images; in the second aspect, when determining the candidate images, the personnel features involved in the comparison are the personnel features of a part of the representative personnel images in the existing image categories, rather than the personnel features of all the personnel images contained in the existing image categories. This can greatly reduce the number of personnel features involved in the comparison when determining the candidate images, thereby reducing the amount of data required to be processed during feature comparison and improving the efficiency of determining the candidate images. Combining the above two aspects, the scheme provided by the embodiment of the present application can greatly improve the efficiency of determining the candidate images, thereby improving the efficiency of image classification based on the candidate images.

[0198] In addition, after determining the candidate image, the solution provided in the embodiment of the present application ultimately determines the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image, that is, the features ultimately used to determine the image category to which the person features to be classified belong are still the first category of person features. Since the first category of person features have a higher feature dimension than the second category of person features and have a stronger ability to identify people, the final image category judgment based on the first category of person features can improve the accuracy of image classification.

[0199] In summary, the overall process of this solution is: based on the comparison between the second category of personnel features with lower feature dimensions, the candidate image is determined from the representative personnel images in the existing classification, and then based on the comparison between the first category of personnel features with higher feature dimensions, it is determined whether the image category to which the image of the person to be classified belongs is the image category to which the candidate image belongs. In this way, based on the above two feature comparisons, the efficiency and accuracy of image classification are taken into account, and the efficiency and accuracy of image classification are significantly improved.

[0200] In one embodiment of the present application, the second-category personnel feature acquisition module 702 is specifically used to input the first-category personnel features of the person image to be classified into a pre-trained feature quantization model to obtain the second-category personnel features of the person image to be classified output by the feature quantization model.

[0201] In this way, by inputting the first type of personnel features of the image of the person to be classified into the pre-trained feature quantization model, the second type of personnel features of the image of the person to be classified output by the feature quantization model can be easily obtained, thereby improving the efficiency of obtaining the second type of personnel features.

[0202] In one embodiment of the present application, the feature quantization model is trained in the following manner: obtaining first-category personnel features of sample personnel images and classification annotations of the sample personnel images; inputting the first-category personnel features of the sample personnel images into a preset neural network model for quantization processing, and obtaining second-category personnel features of the sample personnel images output by the neural network model; inputting the second-category personnel features of the sample personnel images into a pre-trained image classification model for classification, and obtaining classification results of the sample personnel images output by the image classification model; obtaining a first loss value based on the similarity between the first-category personnel features of the sample personnel images and the similarity between the second-category personnel features; obtaining a second loss value based on the classification annotations and classification results of the sample personnel images; adjusting the model parameters of the neural network model based on the first loss value and the second loss value to obtain the feature quantification model.

[0203] It can be seen that, on the first hand, the first loss value is determined based on the similarity between the first category of personnel features and the second category of personnel features of the sample personnel image. Training the neural network model based on the first loss value can enable the model to learn the association between the similarity between the second category of personnel features after quantization of the first category of personnel features of the sample personnel image and the similarity between the original first category of personnel features, thereby ensuring that the similarity between the second category of personnel features after quantization is aligned with the similarity between the first category of personnel features, so that the second category of personnel features after quantization have sufficient distinctiveness; on the second hand, the second loss value is determined based on the classification annotation and classification results of the sample personnel image. Training the neural network model based on the second loss value can enable the model to learn the association between the second low-dimensional features of the sample image and the image category to which the sample personnel image belongs, thereby ensuring that the second category of personnel features after quantization can effectively distinguish personnel images belonging to different image categories in its feature space.

[0204] In summary, after the feature quantization model is obtained through model training based on the first loss value and the second loss value, the second type of personnel features with high discrimination of the image category of the personnel image and aligned with the similarity of the corresponding first type of personnel features can be obtained based on the feature quantization model. That is, more reasonable and accurate second type of personnel features can be obtained, thereby improving the accuracy of subsequent image classification based on the second type of personnel features.

[0205] In one embodiment of the present application, the candidate image determination module 703 includes:

[0206] A similar feature search submodule is used to search for similar features of the second type of personnel features of the person image to be classified in the second type of personnel features included in the attributes of the nodes in the neighbor graph according to the graph structure of the constructed neighbor graph, wherein each node in the neighbor graph corresponds to a representative person image in the existing image category, and the attributes of the node include: the second type of personnel features of the corresponding representative person image, and the relationship between the nodes is established based on the similarity between the second type of personnel features of the representative person images corresponding to the nodes;

[0207] The candidate image determination submodule determines the representative person image corresponding to the node to which the similar feature is searched as the representative person image similar to the person image to be classified as a candidate image.

[0208] As can be seen from the above, when the solution provided by the embodiment of the present application is used to determine the candidate image, the similar features of the second type of personnel features of the image of the person to be classified can be searched in the neighbor graph according to the graph structure of the constructed neighbor graph, and the representative person image corresponding to the node to which the similar features are searched is used as the candidate image. Since the neighbor graph is pre-constructed, the similar features of the second type of personnel features of the image of the person to be classified can be quickly determined based on the relationship between the nodes in the neighbor graph, thereby utilizing the high query rate of the neighbor graph to improve the efficiency of determining the candidate image.

[0209] In addition, on the one hand, the nodes in the neighbor graph correspond to a part of the representative personnel images in the existing image categories, rather than all the personnel images included in the existing image categories, which reduces the number of nodes in the neighbor graph and reduces the storage resource usage of the neighbor graph; on the other hand, since the second type of personnel features are features with lower feature dimensions obtained by compressing the first type of personnel features, the attributes of the nodes in the neighbor graph are set based on the second type of personnel features representing the personnel images, which further reduces the storage resource usage of the neighbor graph.

[0210] In one embodiment of the present application, if the person image to be classified belongs to an existing first image category, the device further includes:

[0211] A judging module, used for judging whether it is necessary to set the person image to be classified as a representative person image of the first image category, and if necessary, triggering an image adding module;

[0212] An image adding module, used for adding the image of the person to be classified to the representative person image of the first image category;

[0213] A neighbor graph updating module is used to update the neighbor graph based on the target node corresponding to the image of the person to be classified.

[0214] It can be seen from the above that when the scheme provided by the embodiment of the present application is used for image classification, when the image of the person to be classified belongs to the existing first image category and it is determined that the image of the person to be classified needs to be set as the representative person image of the first image category, the neighbor graph can be updated in time based on the target node corresponding to the image of the person to be classified. That is, as the image classification scheme proceeds, the neighbor graph can be continuously updated and optimized, further improving the accuracy of subsequent determination of candidate images based on the neighbor graph.

[0215] In one embodiment of the present application, the judgment module is specifically used to obtain the target image quality characterization value of the person image to be classified; judge whether the target image quality characterization value is greater than the minimum image quality characterization value of the representative person image of the first image category; if greater, determine that the person image to be classified needs to be set as the representative person image of the first image category.

[0216] It can be seen that when the target image quality representation value of the person image to be classified is greater than the minimum image quality representation value of the representative person image of the first image category, it can be considered that the image quality of the person image to be classified is high, that is, the representative ability for the image category is strong. Therefore, it can be determined at this time that the person image to be classified needs to be set as the representative person image of the first image category. In this way, the person image to be classified can be set as the representative person image of the first image category in a timely manner when the image quality of the person image to be classified is high, which is conducive to improving the accuracy of subsequent image classification based on the representative person image.

[0217] In one embodiment of the present application, the image adding module is specifically used to determine whether the number of representative personnel images of the first image category is less than a preset number threshold; if not, the representative personnel image corresponding to the minimum image quality characterization value is replaced with the personnel image to be classified; otherwise, the personnel image to be classified is directly added to the representative personnel image of the first image category.

[0218] It can be seen that, in this way, it can be determined to replace the representative person images of the first image category with the person images to be classified or to add the person images to be classified to the representative person images according to the number of representative person images of the first image category. On the one hand, when the number of representative person images is less than the preset number threshold, the person images to be classified can be directly added to the representative person images to ensure that the number of representative person images included in the image category is maintained at the preset number threshold, thereby avoiding too few representative person images included in the image category; on the other hand, when the number of representative person images is not less than the preset number threshold, the representative person images can be replaced with the person images to be classified, so that the existing representative person images can be continuously replaced with images with higher image quality, so that the representative person images are always updated with images with higher image quality, further improving the accuracy of subsequent image classification based on the representative person images.

[0219] In one embodiment of the present application, the preset quantity threshold is determined based on the total number of images included in all existing image categories and is inversely proportional to the total number of images.

[0220] On the one hand, the larger the total number of images mentioned above is, the more image categories there are. Since the representative images of the existing image categories correspond one-to-one to the nodes in the neighbor graph, a smaller preset number threshold is determined in this way, that is, the number of representative images of the existing image categories is reduced, thereby reducing the number of nodes in the neighbor graph, preventing the search efficiency from being too large due to the neighbor graph being too large, which is beneficial to improving the image classification efficiency; conversely, the smaller the total number of images mentioned above is, the fewer the existing image categories are. At this time, a larger preset number threshold is determined, that is, the number of representative images of the existing image categories is increased, thereby increasing the number of nodes in the neighbor graph, preventing the search accuracy from being reduced due to the neighbor graph being too small, which is beneficial to improving the accuracy of image classification.

[0221] In summary, determining a preset number threshold that is inversely proportional to the total number of images can take into account both the efficiency and accuracy of image classification, thereby improving the rationality and practicality of the solution.

[0222] In one embodiment of the present application, if the person image to be classified belongs to a new second image category, the device further includes:

[0223] A node creation module, used to create a target node corresponding to the image of the person to be classified, and set the attributes of the target node to include the second type of person features of the image of the person to be classified;

[0224] A relationship establishment module is used to establish a relationship between the target node and the nodes in the neighbor graph based on the similarity between the second type of personnel features included in the attributes of the target node and the second type of personnel features included in the attributes of the nodes in the neighbor graph.

[0225] It can be seen from the above that when the scheme provided by the embodiment of the present application is used for image classification, when the image of the person to be classified belongs to the new second image category, a target node corresponding to the image of the person to be classified can be created, and the relationship between the target node and the nodes in the neighbor graph can be established, so that the existing neighbor graph can be incrementally updated in a timely manner. That is, as the image classification scheme proceeds, the existing neighbor graph can be continuously incrementally updated, thereby improving the accuracy of subsequent determination of candidate images based on the neighbor graph.

[0226] In one embodiment of the present application, the image category determination module 704 is specifically used to calculate the similarity between the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image; determine whether the maximum value of the calculated similarities is greater than a preset similarity threshold; if yes, determine that the image category to which the person image to be classified belongs is the image category to which the target image belongs, wherein the target image is: the candidate image to which the first category of personnel features corresponding to the maximum value belong; if no, determine that the person image to be classified belongs to a new image category, and determine the person image to be classified as a representative person image of the new image category.

[0227] It can be seen that first, the target image with the greatest similarity between the corresponding first-category personnel feature and the first-category personnel feature of the person image to be classified is determined from the candidate images, and then, in combination with the preset similarity threshold, when the above similarity is greater than the similarity threshold, the image category to which the person image to be classified belongs is determined as the image category to which the target image belongs. In this way, on the one hand, it can be ensured that the image category to which the person image to be classified belongs can be determined as the image category to which the target image with the greatest similarity between the corresponding first-category personnel feature and the first-category personnel feature of the person image to be classified belongs, which is conducive to improving the accuracy of image classification; on the other hand, it is further ensured based on the preset similarity threshold that the above similarity needs to meet certain requirements, further improving the accuracy of image classification.

[0228] In the solutions provided in the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all carried out under the premise of the user's knowledge and authorization, and comply with the relevant laws and regulations and do not violate public order and good morals.

[0229] It should be noted that the two-dimensional face images in this embodiment come from a public data set.

[0230] The present application also provides an electronic device, such as Figure 8 As shown, including:

[0231] Memory 801, used for storing computer programs;

[0232] The processor 802 is used to implement the aforementioned image classification method when executing the program stored in the memory 801.

[0233] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0234] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0235] The communication interface is used for communication between the above electronic device and other devices.

[0236] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0237] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0238] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned image classification method are implemented.

[0239] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image classification method in the above embodiments.

[0240] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0241] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0242] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0243] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image classification method, characterized in that: The method comprises: Obtaining first-category person features of the person image to be classified; Performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified; Based on the second category of personnel features of the person image to be classified and the second category of personnel features of representative person images in existing image categories, determining a representative person image similar to the person image to be classified as a candidate image; Based on the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image, the image category to which the person image to be classified belongs is determined.

2. The method according to claim 1, characterized in that: The performing feature dimensionality reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified includes: The first type of personnel features of the person image to be classified are input into a pre-trained feature quantization model to obtain the second type of personnel features of the person image to be classified output by the feature quantization model.

3. The method according to claim 2, characterized in that The feature quantization model is trained in the following way: Obtaining first-category personnel features of sample personnel images and classification annotations of sample personnel images; Inputting the first type of personnel features of the sample personnel image into a preset neural network model for quantization processing, and obtaining the second type of personnel features of the sample personnel image output by the neural network model; Inputting the second type of personnel features of the sample personnel image into a pre-trained image classification model for classification, and obtaining a classification result of the sample personnel image output by the image classification model; Obtaining a first loss value based on the similarity between the first type of personnel features and the similarity between the second type of personnel features of the sample personnel images; Obtaining a second loss value based on the classification annotation and classification results of the sample person image; Based on the first loss value and the second loss value, the model parameters of the neural network model are adjusted to obtain the feature quantization model.

4. The method according to claim 1, characterized in that: The determining, based on the second category of personnel features of the personnel image to be classified and the second category of personnel features of representative personnel images in existing image categories, a representative personnel image similar to the personnel image to be classified as a candidate image, comprises: According to the graph structure of the constructed neighbor graph, in the second type of personnel features included in the attributes of the nodes in the neighbor graph, similar features of the second type of personnel features of the person image to be classified are searched, wherein each node in the neighbor graph corresponds to a representative person image in the existing image category, and the attributes of the node include: the second type of personnel features of the corresponding representative person image, and the relationship between the nodes is established based on the similarity between the second type of personnel features of the representative person images corresponding to the nodes; The representative person image corresponding to the node to which the similar feature is searched is determined to be a representative person image similar to the person image to be classified, and is taken as a candidate image.

5. The method according to claim 4, characterized in that If the person image to be classified belongs to an existing first image category, the method further includes: Determining whether it is necessary to set the to-be-classified person image as a representative person image of the first image category; If necessary, adding the image of the person to be classified to the representative person images of the first image category; Based on the target node corresponding to the person image to be classified, the neighbor graph is updated.

6. The method according to claim 5, characterized in that The determining whether it is necessary to set the to-be-classified person image as a representative person image of the first image category includes: Obtaining a target image quality representation value of the person image to be classified; Determining whether the target image quality representation value is greater than a minimum image quality representation value of representative person images of the first image category; If it is greater, it is determined that the person image to be classified needs to be set as a representative person image of the first image category.

7. The method according to claim 5, characterized in that The step of adding the person image to be classified to the representative person image of the first image category includes: Determining whether the number of representative person images of the first image category is less than a preset number threshold; If it is not less than, replacing the representative person image corresponding to the minimum image quality representation value with the person image to be classified; Otherwise, the person image to be classified is directly added to the representative person image of the first image category.

8. The method according to claim 7, characterized in that The preset quantity threshold is determined based on the total number of images included in all existing image categories and is inversely proportional to the total number of images.

9. The method according to claim 4, characterized in that If the person image to be classified belongs to a new second image category, the method further includes: Creating a target node corresponding to the image of the person to be classified, and setting the attributes of the target node to include the second type of person features of the image of the person to be classified; Based on the similarity between the second type of personnel features included in the attributes of the target node and the second type of personnel features included in the attributes of the nodes in the neighbor graph, a relationship between the target node and the nodes in the neighbor graph is established.

10. The method according to any one of claims 1 to 9, characterized in that The determining the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image comprises: Calculating the similarity between the first category of personnel features of the to-be-classified personnel image and the first category of personnel features of the candidate image; Determine whether the maximum value among the calculated similarities is greater than a preset similarity threshold; If yes, determine that the image category to which the to-be-classified person image belongs is the image category to which the target image belongs, wherein the target image is: a candidate image to which the first category of person features corresponding to the maximum value belongs; If not, it is determined that the person image to be classified belongs to a new image category, and the person image to be classified is determined as a representative person image of the new image category.

11. An image classification device, characterized in that: The device comprises: A first-category personnel feature acquisition module, used to acquire first-category personnel features of the personnel image to be classified; A second type of personnel feature acquisition module, used for performing feature dimension reduction processing on the first type of personnel features of the personnel image to be classified to obtain the second type of personnel features of the personnel image to be classified; A candidate image determination module, configured to determine, based on the second category of personnel features of the personnel image to be classified and the second category of personnel features of representative personnel images in existing image categories, a representative personnel image similar to the personnel image to be classified as a candidate image; The image category determination module is used to determine the image category to which the person image to be classified belongs based on the first category of person features of the person image to be classified and the first category of person features of the candidate image.

12. The device according to claim 11, characterized in that The second-category personnel feature acquisition module is specifically used to input the first-category personnel features of the to-be-classified personnel image into a pre-trained feature quantization model to obtain the second-category personnel features of the to-be-classified personnel image output by the feature quantization model; or The feature quantization model is trained in the following manner: obtaining first-category personnel features of sample personnel images and classification annotations of sample personnel images; inputting the first-category personnel features of the sample personnel images into a preset neural network model for quantization processing, and obtaining second-category personnel features of the sample personnel images output by the neural network model; Inputting the second type of personnel features of the sample personnel images into a pre-trained image classification model for classification, and obtaining a classification result of the sample personnel images output by the image classification model; obtaining a first loss value based on the similarity between the first type of personnel features and the similarity between the second type of personnel features of the sample personnel images; obtaining a second loss value based on the classification annotations and classification results of the sample personnel images; adjusting the model parameters of the neural network model based on the first loss value and the second loss value, and obtaining the feature quantization model; or The candidate image determination module includes: a similar feature search submodule, which is used to search for similar features of the second type of personnel features of the person image to be classified in the second type of personnel features included in the attributes of the nodes in the neighbor graph according to the graph structure of the constructed neighbor graph, wherein each node in the neighbor graph corresponds to a representative person image in the existing image category, and the attributes of the node include: the second type of personnel features of the corresponding representative person image, and the relationship between the nodes is established based on the similarity between the second type of personnel features of the representative person images corresponding to the nodes; the candidate image determination submodule, which determines the representative person image corresponding to the node to which the similar feature is searched as a representative person image similar to the person image to be classified as a candidate image; or If the person image to be classified belongs to an existing first image category, the device further includes: a judgment module, used to judge whether it is necessary to set the person image to be classified as a representative person image of the first image category, and if necessary, trigger an image adding module; an image adding module, used to add the person image to be classified to the representative person image of the first image category; a neighbor graph updating module, used to update the neighbor graph based on the target node corresponding to the person image to be classified; or The judgment module is specifically used to obtain a target image quality representation value of the person image to be classified; judge whether the target image quality representation value is greater than a minimum image quality representation value of a representative person image of the first image category; if greater, determine that the person image to be classified needs to be set as a representative person image of the first image category; or The image adding module is specifically used to determine whether the number of representative person images of the first image category is less than a preset number threshold; if not, replace the representative person image corresponding to the minimum image quality representation value with the person image to be classified; otherwise, directly add the person image to be classified to the representative person images of the first image category; or The preset number threshold is determined based on the total number of images included in all existing image categories and is inversely proportional to the total number of images; or If the person image to be classified belongs to a new second image category, the device further includes: a node creation module, used to create a target node corresponding to the person image to be classified, and set the attributes of the target node to include the second type of person features of the person image to be classified; a relationship establishment module, used to establish a relationship between the target node and the nodes in the neighbor graph based on the similarity between the second type of person features included in the attributes of the target node and the second type of person features included in the attributes of the nodes in the neighbor graph; or The image category determination module is specifically used to calculate the similarity between the first category of personnel features of the person image to be classified and the first category of personnel features of the candidate image; determine whether the maximum value of the calculated similarities is greater than a preset similarity threshold; if so, determine that the image category to which the person image to be classified belongs is the image category to which the target image belongs, wherein the target image is: the candidate image to which the first category of personnel features corresponding to the maximum value belong; if not, determine that the person image to be classified belongs to a new image category, and determine the person image to be classified as a representative person image of the new image category.

13. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-10 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images