Image ranking model, method, device and electronic equipment

Abstract

Embodiments of the present application disclose an image grade classification model, method, device and electronic equipment, which are applied to the technical field of appearance grade classification and can solve the problem of how to realize accurate grade classification based on images. The classification model comprises a recommended classification network and a twin classification network, and the recommended classification network and the twin classification network are connected. The recommended classification network is used for extracting first image features of an image and determining a recommended classification result of the image according to the first image features. The twin classification network is used for determining a matching degree of the image and a target standard image according to the recommended classification result and outputting a grade classification result of the image according to the matching degree. When the matching degree is greater than a threshold value, the recommended classification result is output, and the target standard image is a standard image corresponding to the recommended classification result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of hierarchical classification technology, and in particular to an image hierarchical classification model, method, apparatus and electronic device. Background Technology

[0002] With the advancement of technology, the demand for the accuracy of appearance grade inspection of electronic products is constantly increasing. At present, appearance grade classification algorithms mainly use various methods such as traditional image feature extraction, feature refinement and quantization, feature binary tree classification, and deep learning classification networks. However, they generally suffer from problems such as being sensitive to image parameters or the classification network being unable to refine feature grades. This results in the low accuracy of current appearance grade classification. Therefore, how to achieve accurate grade classification based on images has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides an image grading model, method, apparatus, and electronic device to solve the problem of how to achieve accurate image-based grading in the prior art.

[0004] Firstly, a hierarchical classification model for images is provided, the hierarchical classification model including a recommendation classification network and a Siamese classification network, wherein the recommendation classification network and the Siamese classification network are connected, wherein...

[0005] The recommendation classification network is used to extract the first image features of the image and determine the recommendation classification result of the image based on the first image features;

[0006] The Siamese classification network is used to determine the matching degree between the image and the target standard image based on the recommended classification result, and output the level classification result of the image based on the matching degree; wherein when the matching degree is greater than a threshold, the recommended classification result is output, and the target standard image is the standard image corresponding to the recommended classification result.

[0007] As an optional implementation, in a first aspect of the present invention, the recommendation classification network includes a feature extraction module, a feature classification module, and a first output module. The feature extraction module is connected to the feature classification module, and the feature classification module is connected to the first output module.

[0008] The feature extraction module is used to extract the first image features from the image;

[0009] The feature classification module is used to classify the first image features using a region feature classification method to obtain the feature classification result.

[0010] The first output module is used to output the recommended classification result based on the feature classification result.

[0011] As an optional implementation, in a first aspect of the present invention, the Siamese classification network includes a feature comparison module, a feature matching module, and a second output module. The feature comparison module is connected to the feature matching module, and the feature matching module is connected to the second output module.

[0012] The feature comparison module is used to obtain the target standard image based on the recommended classification result, and compare the first image feature of the image with the second image feature of the target standard image to obtain a comparison result;

[0013] The feature matching module is used to determine the matching degree based on the comparison result;

[0014] The second output module is used to output the level classification result of the image based on the matching degree.

[0015] As an optional implementation, in a first aspect of the present invention, the second output module is specifically configured to output the recommended classification result when the matching degree is greater than or equal to the threshold; and to output an abnormal prompt message when the matching degree is less than the threshold, the abnormal prompt message being used to indicate that there is an abnormality in the level classification of the image.

[0016] As an optional implementation, in a first aspect of the present invention, the hierarchical classification model further includes a feature sharing module, which is connected to both the feature extraction module and the feature comparison module, wherein...

[0017] The feature sharing module is used to obtain the first image features from the feature extraction module and send the first image features to the feature comparison module.

[0018] As an optional implementation, in a first aspect of the present invention, when the number of recommended classification results is at least two, the hierarchical classification model further includes a probability flexibility maximum output module, which is connected to both the first output module and the feature comparison module.

[0019] The probability flexibility maximum output module is used to sort multiple recommendation classification results according to a preset order and output at least two sorted recommendation classification results to the feature comparison module.

[0020] The feature comparison module is specifically used to obtain the target standard image based on one of the sorted recommended classification results, and to compare the first image feature of the image with the second image feature of the target standard image to obtain the comparison result, wherein one of the sorted recommended classification results is selected according to the preset order.

[0021] As an optional implementation, in a first aspect of the present invention, the hierarchical classification model further includes an iteration count determination module, which is connected to both the second output module and the feature comparison module.

[0022] The iteration count determination module is used to record the number of times the matching degree is less than the threshold. When the number is less than or equal to a preset number, it instructs the feature comparison module to obtain the updated target standard image in the preset order, so that the second output module outputs the image's level classification result based on the updated target standard image. The updated target standard image is determined based on other sorted recommended classification results besides one of the sorted recommended classification results.

[0023] Secondly, a method for classifying the hierarchy of images is provided, the method comprising: acquiring an image of an object to be tested;

[0024] The image is input into an image classification model to obtain the image classification result;

[0025] The hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network extracts a first image feature from the image and determines a recommendation classification result for the image based on the first image feature. The Siamese classification network determines the matching degree between the image and a target standard image based on the recommendation classification result and outputs the hierarchical classification result of the image based on the matching degree. When the matching degree is greater than a threshold, the recommendation classification result is output, and the target standard image is the standard image corresponding to the recommendation classification result.

[0026] As an optional implementation, in a second aspect of the present invention, before inputting the image into an image hierarchical classification model to obtain the image hierarchical classification result, the method further includes:

[0027] Acquire multiple training data pairs, each training data pair including a training image and the training level classification result corresponding to the training image;

[0028] The model is trained based on the multiple training data sets to obtain the hierarchical classification model of the image.

[0029] As an optional implementation, in a second aspect of the present invention, the step of training the model based on the plurality of training data pairs to obtain the hierarchical classification model of the image includes:

[0030] The recommended training network is trained based on the multiple training data.

[0031] When the recommendation training network is in a convergent state, the recommendation classification network is obtained;

[0032] The Siamese training network is trained based on the multiple training data pairs and the recommendation classification network.

[0033] When the twin training network is in a convergent state, the twin classification network is obtained.

[0034] Thirdly, an image classification device is provided, the classification device comprising: an acquisition module for acquiring an image of an object to be tested;

[0035] The grading module is used to input the image into the image grading model to obtain the grading result of the image;

[0036] The hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network extracts a first image feature from the image and determines a recommendation classification result for the image based on the first image feature. The Siamese classification network determines the matching degree between the image and a target standard image based on the recommendation classification result and outputs the hierarchical classification result of the image based on the matching degree. When the matching degree is greater than a threshold, the recommendation classification result is output, and the target standard image is the standard image corresponding to the recommendation classification result.

[0037] Fourthly, an electronic device is provided, the electronic device comprising:

[0038] Memory containing executable program code;

[0039] A processor coupled to the memory;

[0040] The processor calls the executable program code stored in the memory to execute the image level classification method in the second aspect of the present invention.

[0041] Fifthly, a computer-readable storage medium is provided that stores a computer program that causes a computer to execute the image grading method of the second aspect of the present invention. The computer-readable storage medium includes ROM / RAM, a magnetic disk, or an optical disk, etc.

[0042] In a sixth aspect, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform some or all of the steps of any of the methods of the second aspect.

[0043] In a seventh aspect, an application publishing platform is provided for publishing computer program products, wherein when the computer program products are run on a computer, the computer performs some or all of the steps of any of the methods of the second aspect.

[0044] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0045] In this embodiment of the invention, the image hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network extracts first image features and determines a recommended classification result based on these features. The Siamese classification network determines the matching degree between the image and a target standard image based on the recommended classification result and outputs the hierarchical classification result based on the matching degree. When the matching degree is greater than a threshold, a recommended classification result is output, and the target standard image is the standard image corresponding to the recommended classification result. This hierarchical classification model integrates the recommendation classification network and the Siamese classification network. Based on the recommended classification result obtained from the recommendation classification network, the result is matched again through the Siamese classification network, effectively improving the accuracy of the hierarchical classification model. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a structural schematic diagram of an image hierarchical classification model provided in an embodiment of the present invention. Figure 1 ;

[0048] Figure 2 This is a structural schematic diagram of an image hierarchical classification model provided in an embodiment of the present invention. Figure 2 ;

[0049] Figure 3 This is a structural schematic diagram of an image hierarchical classification model provided in an embodiment of the present invention. Figure 3 ;

[0050] Figure 4 This is a flowchart illustrating an image hierarchical classification method provided in an embodiment of the present invention. Figure 1 ;

[0051] Figure 5 This is a flowchart illustrating an image hierarchical classification method provided in an embodiment of the present invention. Figure 2 ;

[0052] Figure 6 This is a flowchart illustrating an image hierarchical classification method provided in an embodiment of the present invention. Figure 3 ;

[0053] Figure 7 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present invention. Figure 1 ;

[0054] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present invention. Figure 2 ;

[0055] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The terms "first" and "second," etc., used in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order of objects. For example, "first image feature" and "second image feature," etc., are used to distinguish different image features, not to describe a specific order of image features.

[0058] The terms “comprising” and “having” and any variations thereof in this invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0059] It should be noted that in the embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] In related technologies, the classification of appearance levels for electronic products generally employs classification algorithms, typically including two implementation methods: traditional image feature classification and deep learning classification. Traditional image feature classification utilizes traditional image feature extraction, feature refinement and quantization, and feature binary tree classification. Traditional algorithms are extremely sensitive to image grayscale, making them easily affected by changes in image feature grayscale, resulting in a lack of generalizability in classification performance. Furthermore, traditional image processing methods have complex parameters, are highly specific, and are difficult to port and replicate, presenting numerous challenges in algorithmic and engineering deployment. While deep learning classification networks can reduce the impact of image grayscale changes on the results, a single classification network struggles to refine feature levels. Single-classification networks focus on image-level classification or semantic features, but they fail to extract regional feature bases for similar semantic level classifications, thus resulting in low classification accuracy.

[0061] To solve the above problems, such as Figure 1 As shown, this embodiment of the invention provides an image hierarchical classification model 10, which may include a recommendation classification network 11 and a twin classification network 12, which are interconnected.

[0062] The recommendation classification network 11 can be used to extract the first image features of the image and determine the recommendation classification result of the image based on the first image features.

[0063] It should be noted that the image can be an image of the appearance of the object under test, or an image of other parts of the object under test; the embodiments of the present invention do not impose specific limitations.

[0064] Optionally, the recommended classification network 11 is an auxiliary classification network based on the learning of regional features of the label. It adopts a randomized regional approach to add an auxiliary learning branch that forces the learning of regional features, so that the classification network pays more attention to regional features, thereby improving the classification success rate and reducing the overkill rate of hierarchical classification.

[0065] The Siamese classification network 12 can be used to determine the matching degree between the image and the target standard image based on the recommended classification result, and output the image's hierarchical classification result based on the matching degree; when the matching degree is greater than a threshold, the recommended classification result is output, and the target standard image is the standard image corresponding to the recommended classification result.

[0066] It should be noted that the image classification results can be used to characterize the appearance grade of the object under test. This appearance grade has a significant impact on the quality of the object. For example, an industrial metal product may be classified into four appearance grades: A, B, C, and D. Grade A is considered superior; Grade B has minor defects and does not require rework; Grade C has some defects but does not affect functionality and requires rework; Grade D has severe defects that affect quality and function and must be discarded. Therefore, the product quality of the object under test can be determined based on the classification results.

[0067] Optionally, after calculating the matching degree, the Siamese classification network 12 can output the corresponding level classification result based on the matching degree. Furthermore, when the matching degree is greater than a threshold, the Siamese classification network 12 can output the recommended classification result, which means that when the matching degree is greater than the threshold, the level classification result is the recommended classification result.

[0068] Optionally, when calculating the matching degree, the twin classification network 12 compares the image with the target standard image. The target standard image can be a standard image pre-stored by the twin classification network 12, or it can be obtained by the twin classification network 12 from the cloud server in real time. This embodiment of the invention does not make specific limitations.

[0069] Optional, such as Figure 2 As shown, the recommendation classification network 11 may specifically include: a feature extraction module 111, a feature classification module 112, and a first output module 113, wherein the feature extraction module 111 is connected to the feature classification module 112, and the feature classification module 112 is connected to the first output module 113.

[0070] The feature extraction module 111 is used to extract first image features from the image.

[0071] It should be noted that the feature extraction module 111 may be equipped with a feature extractor, which can be used to extract features from the image to obtain the first image features of the image.

[0072] Optionally, the first image feature may include feature pixels, feature parameters, etc. of the image, and the feature parameters may include grayscale value, pixel value, balance value, etc.

[0073] The feature classification module 112 is used to classify the features of the first image using a region feature classification method to obtain the feature classification result.

[0074] It should be noted that the feature classification module 112 can perform feature classification on the first image features after extracting the first image features from the image, and analyze the first image features to obtain the feature classification result.

[0075] Optionally, the feature classification module 112 pre-stores the correspondence between relevant image features and classification results, so that the feature classification module 112 can determine the feature classification result corresponding to the first image feature after obtaining the first image feature.

[0076] It should be noted that the feature classification result refers to the level classification corresponding to each first image feature. Since there are multiple first image features in an image, the level corresponding to each first image feature may be different. The feature classification module 112 can classify each first image feature according to the level. For example, suppose there are five levels: A, B, C, D, and E. The feature extraction module 111 extracts 1000 first image features from the image. At this time, the feature classification module 112 can classify these 1000 first image features according to the correspondence between the image features and the classification results, that is, determine which level each first image feature belongs to. The resulting feature classification result may be: 830 first image features belong to level A, 88 first image features belong to level B, 70 first image features belong to level C, 10 first image features belong to level D, and 2 first image features belong to level E.

[0077] The correspondence between the image features and the classification results can be obtained through pre-trained model training and stored in the feature classification module 112.

[0078] The first output module 113 is used to output recommended classification results based on the feature classification results.

[0079] It should be noted that in the first output module 113, the feature classification result can be converted into a recommended classification result and the recommended classification result can be output.

[0080] The recommended classification result may include the level corresponding to the image and the probability corresponding to the level. The number of levels may be one or more, which is determined based on the feature classification result; and the sum of the probabilities corresponding to each level is 1.

[0081] For example, suppose there are five levels, namely A, B, C, D and E. The feature extraction module 111 extracts 1000 first image features from the image. At this time, the feature classification module 112 can classify these 1000 first image features according to the correspondence between the image features and the classification results, that is, determine which level each first image feature belongs to. Then the feature classification results obtained by the feature classification module 112 can be: 830 first image features belong to level A, 88 first image features belong to level B, 70 first image features belong to level C, 10 first image features belong to level D, and 2 first image features belong to level E. Then, the first output module 113 can process the feature classification results to obtain the recommended classification results, namely, the probability of level A is 830 / 1000 = 0.83, the probability of level B is 88 / 1000 = 0.088, the probability of level C is 70 / 1000 = 0.07, the probability of level D is 10 / 1000 = 0.01, and the probability of level E is 2 / 1000 = 0.002.

[0082] Optional, such as Figure 2 As shown, the twin classification network 12 may specifically include: a feature comparison module 121, a feature matching module 122, and a second output module 123, wherein the feature comparison module 121 is connected to the feature matching module 122, and the feature matching module 122 is connected to the second output module 123.

[0083] The feature comparison module 121 is used to obtain the target standard image based on the recommended classification result, and compare the first image feature of the image with the second image feature of the target standard image to obtain the comparison result.

[0084] It should be noted that the target standard image is the standard image corresponding to the recommended classification result. Since the recommended classification result includes at least one level and the probability corresponding to that level, the target standard image is the standard image corresponding to each level. That is, the number of target standard images is related to the number of levels included in the recommended classification result. For example, if the recommended classification result of the image is that the probability corresponding to level A is 0.91, the probability corresponding to level B is 0.08, and the probability corresponding to level C is 0.01, then the feature comparison module 121 needs to obtain the standard image corresponding to level A, the standard image corresponding to level B, and the standard image corresponding to level C.

[0085] Optionally, after the feature comparison module 121 acquires the target standard image, it needs to perform a similarity comparison between the image and the target standard image. In the process of comparison, it is actually comparing the first image feature of the image and the second image feature of the target standard image. That is, after acquiring the target standard image, the feature comparison module 121 needs to extract features from the target standard image, that is, extract the second image feature from the target standard image, and then compare the first image feature and the second image feature to obtain the comparison result.

[0086] Optionally, since the recommended classification result can include at least one level, when comparing the image and the target standard image, the image is compared with the target standard image corresponding to each level separately. For example, assuming that the recommended classification result of the image includes level A, level B and level C, the feature comparison module 121 can first compare the image with the standard image corresponding to level A to obtain the matching degree; then compare the image with the standard image corresponding to level B to obtain the matching degree; and finally compare the image with the standard image corresponding to level C to obtain the matching degree. Of course, the order of comparison can be changed.

[0087] The feature matching module 122 is used to determine the matching degree based on the comparison results.

[0088] It should be noted that the feature matching module 122 can be used to convert the comparison result into a matching degree. The comparison result only describes the difference between each image feature, while the matching degree is a summary of the comparison result, that is, it can be used to characterize the similarity between the first image feature and the second image feature, that is, the similarity between the image and the target standard image.

[0089] The second output module 123 is used to output the image's classification results based on the matching degree.

[0090] Optionally, the second output module 123 can be used to output a recommended classification result when the matching degree is greater than or equal to the threshold; and to output an abnormal prompt message when the matching degree is less than the threshold. The abnormal prompt message is used to indicate that there is an abnormality in the hierarchical classification model for the hierarchical classification of the image.

[0091] The threshold can be arbitrarily set by the second output module 123, or it can be set by the second output module 123 based on historical detection data, or it can be set by staff. This embodiment of the invention does not impose any specific limitations.

[0092] It should be noted that if the matching degree is greater than or equal to the threshold, it means that the image and the target standard image are quite similar, and the image can be determined to be the level corresponding to the target standard image. In this case, the second output module 123 can directly output the recommended classification result. If the matching degree is less than the threshold, it means that the image and the target standard image are not similar, and the image can be determined not to be the level corresponding to the target standard image. In other words, it is possible that the recommended classification network 11 has an anomaly in the feature classification of the image, or that the Siamese classification network 12 has an anomaly in the comparison between the image and the target standard image. Therefore, the second output module 123 can output an anomaly prompt message so that the staff can perform fault detection and troubleshooting on the level classification model 10 based on the output anomaly prompt message.

[0093] Optionally, when determining the matching degree between the image and the target standard image and the output result, the Siamese classification network 12 can sort according to the probability of each level or according to the matching degree.

[0094] It should be noted that when the recommended classification result only includes a level and the probability corresponding to that level, the Siamese classification network 12 only needs to compare the image with the target standard image corresponding to that level, obtain the matching degree, and then output the corresponding result based on the matching degree.

[0095] For example, assuming the threshold is 0.7, the recommended classification result of the image only includes level A, and the probability corresponding to level A is 1. Then, the feature comparison module 121 can first compare the image with the standard image corresponding to level A to obtain the comparison result. Then, if the feature matching module 122 obtains a matching degree of 0.9 based on the comparison result, since 0.9 is greater than 0.7, the second output module 123 can output the level classification result, that is, the level corresponding to the image is level A. If the feature matching module 122 obtains a matching degree of 0.5 based on the comparison result, since 0.5 is less than 0.7, the second output module 123 can output an abnormal prompt message.

[0096] It should be noted that when the recommended classification result includes at least two levels and the probability corresponding to each level, the Siamese classification network 12 can specifically include the following implementation methods:

[0097] Implementation Method 1: The Siamese classification network 12 compares the image with the target standard image corresponding to each level in order of the probability of each level. When the matching degree of a certain level is greater than or equal to the threshold, the Siamese classification network 12 can directly output that level as the level classification result. When all levels have been compared and there is still no matching degree greater than or equal to the threshold, the Siamese classification network 12 can output an abnormal prompt message.

[0098] For example, assuming a threshold of 0.7, the probability of an image being classified as grade A is 0.91, grade B is 0.08, and grade C is 0.01. Sorted in ascending order, these are grade A, grade B, and grade C. The feature comparison module 121 first compares the image with the standard image corresponding to grade A to obtain a comparison result. If the feature matching module 122 obtains a matching degree of 0.9 based on this comparison result, since 0.9 is greater than 0.7, the second output module 123 can output the grade classification result, i.e., the image corresponds to grade A. If the feature matching module 122 obtains a matching degree of 0.5 based on this comparison result, since 0.5 is less than 0.7, the feature comparison module 121 then compares the image with the standard image corresponding to grade B to obtain a comparison result. If the feature matching module 122 obtains a matching degree of 0.75 based on this comparison result, since 0.75 is greater than 0.7, the second output module 123... The output classification result is B, indicating that the image corresponds to level B. If the feature matching module 122 obtains a matching degree of 0.68 based on the comparison result, since 0.68 is less than 0.7, the feature comparison module 121 can then compare the image with the standard image corresponding to level C to obtain a comparison result. If the feature matching module 122 obtains a matching degree of 0.95 based on the comparison result, since 0.95 is greater than 0.7, the second output module 123 can output the classification result, indicating that the image corresponds to level C. If the feature matching module 122 obtains a matching degree of 0.44 based on the comparison result, since 0.44 is less than 0.7, the second output module 123 can output an abnormal prompt message.

[0099] Implementation Method 2: The Siamese classification network 12 can compare the image with the target standard image corresponding to each level to obtain the matching degree corresponding to each level. Then, each matching degree is compared with a threshold, and the level corresponding to the largest matching degree among all those greater than or equal to the threshold is determined as the level classification result, which is output by the Siamese classification network 12. If all matching degrees are less than the threshold, the Siamese classification network 12 can output an abnormal prompt message.

[0100] For example, assuming a threshold of 0.7, the probability of an image being classified as grade A is 0.91, grade B is 0.08, and grade C is 0.01. The feature comparison module 121 compares the image with the standard image corresponding to grade A to obtain a first comparison result, then compares the image with the standard image corresponding to grade B to obtain a second comparison result, and finally compares the image with the standard image corresponding to grade C to obtain a third comparison result. Then, the feature matching module 122 obtains a first matching degree of 0.75 based on the first comparison result, a second matching degree of 0.89 based on the second comparison result, and a third matching degree of 0.53 based on the third comparison result. Comparing these three matching degrees with the threshold, we find that 0.75 is greater than 0.7, 0.89 is greater than 0.7, and 0.53 is less than 0.7, and 0.89 is greater than 0.75. Therefore, the second output module 123 can determine grade B corresponding to the second matching degree as the grade classification result and output it.

[0101] Optional, such as Figure 3 As shown, the hierarchical classification model 10 may also include a feature sharing module 13, which is connected to the feature extraction module 111 and the feature comparison module 121 respectively.

[0102] The feature sharing module 13 is used to obtain the first image features from the feature extraction module and send the first image features to the feature comparison module.

[0103] It should be noted that in this hierarchical classification model 10, the recommendation classification network 11 and the twin classification network 12 can share features through the feature sharing module 13. That is, the first image features of the image extracted by the recommendation classification network 11 can be shared with the twin classification network 12 through the feature sharing module 13, so that the twin classification network 12 no longer needs to repeatedly extract the first image features of the image.

[0104] Optional, such as Figure 3 As shown, the hierarchical classification model 10 may also include a probabilistic flexible maximum output module 14, which is connected to the first output module 113 and the feature comparison module 121 respectively.

[0105] The probability flexibility maximum output module 14 is used to sort multiple recommended classification results in a preset order and output at least two sorted recommended classification results to the feature comparison module 121.

[0106] It should be noted that when the number of recommended classification results is at least two, that is, when the recommended classification network 11 outputs at least two levels and the probability corresponding to each level, the probability flexibility maximum output module 14 can sort the at least two recommended classification results. The preset order can be from largest to smallest or from smallest to largest, and then output to the feature comparison module 121 in the Siamese classification network 12.

[0107] Optionally, in this implementation, the feature comparison module 121 is specifically used to obtain the target standard image based on one of the sorted recommended classification results, and to compare the first image features of the image with the second image features of the target standard image to obtain a comparison result, wherein one of the sorted recommended classification results is selected according to a preset order.

[0108] It should be noted that in this implementation, the feature comparison module 121 can select one of the recommended classification results according to the preset order to obtain the corresponding target standard image. The recommended classification result can be the level with the highest probability or the level with the lowest probability.

[0109] Optional, such as Figure 3 As shown, the hierarchical classification model 10 may further include an iteration number determination module 15, which is connected to the second output module 123 and the feature comparison module 121 respectively.

[0110] The iteration count judgment module 15 is used to record the number of times the matching degree is less than the threshold. When the number is less than or equal to the preset number, it instructs the feature comparison module 121 to obtain the updated target standard image in a preset order, so that the second output module 123 outputs the image level classification result based on the updated target standard image. The updated target standard image is determined based on the other sorted recommended classification results except for one of the sorted recommended classification results.

[0111] It should be noted that when the matching degree is less than the threshold, the feature comparison module 121 needs to re-compare the image with the target standard image corresponding to other levels. Therefore, the iteration number judgment module 15 can record the number of times the matching degree is less than the threshold.

[0112] If the number of iterations is less than or equal to the preset number of iterations, the iteration number determination module 15 can instruct the feature comparison module 121 to select a target standard image corresponding to another level for comparison with the image; if the number of iterations is greater than the preset number of iterations, the iteration number determination module 15 can directly instruct the second output module 123 to output an abnormal prompt message.

[0113] For example, assuming a threshold of 0.7 and a preset number of iterations of 2, the recommended classification results for the image are: probability of 0.85 for level A, probability of 0.08 for level B, probability of 0.01 for level C, and probability of 0.06 for level D. The order of probability from highest to lowest is level A, level B, level D, and level C. The feature comparison module 121 can compare the image with the standard image corresponding to level A to obtain a first comparison result. Then, the feature matching module 122 obtains a first matching degree of 0.56 based on the first comparison result. Since 0.56 is less than 0.7, the iteration count judgment module 15 records one iteration. Because 1 is less than the preset number of iterations of 2, the iteration count judgment module 15 can instruct the feature comparison module 121 to compare the image with the standard image corresponding to level B to obtain a second comparison result. Then, the feature matching module 122 obtains a second matching degree of 0.12 based on the second comparison result. If 0.12 is less than 0.7, then the iteration count judgment module 15 can record it twice. Since 2 equals the preset count 2, the iteration count judgment module 15 can instruct the feature comparison module 121 to compare the image with the standard image corresponding to level D and obtain the third comparison result. Then, the feature matching module 122 obtains the third matching degree as 0.66 based on the third comparison result. Since 0.66 is less than 0.7, the iteration count judgment module 15 can record it three times. Since 3 is greater than the preset count 2, the iteration count judgment module 15 can instruct the second output module 123 to output an abnormal prompt message.

[0114] In this implementation, the iteration count determination module can avoid the hierarchical classification model from repeatedly comparing the image and the target standard image indefinitely. After a certain number of iterations, the corresponding result can be directly output, which reduces the workload and power consumption of the hierarchical classification model and improves the efficiency of the hierarchical classification model for hierarchical classification.

[0115] like Figure 4 As shown, this embodiment of the invention provides a method for image hierarchical classification. This method can be applied to an image hierarchical classification device, which may be equipped with the aforementioned image hierarchical classification model 10. The hierarchical classification method may include the following steps:

[0116] 401. Obtain the image of the object to be tested.

[0117] In this embodiment of the invention, the classification device can acquire the image, which can be obtained by itself through a camera, uploaded by staff, or obtained from a cloud server.

[0118] 402. Input the image into the image classification model to obtain the image classification result.

[0119] In this embodiment of the invention, the classification device can input the image into the classification model so that the classification model can classify the image into a classification level, thereby obtaining the classification result of the image.

[0120] Among them, such as Figure 1 As shown, the hierarchical classification model 10 may include a recommendation classification network 11 and a Siamese classification network 12, which are interconnected. After the hierarchical classification device inputs an image into the hierarchical classification model, the recommendation classification network 11 first extracts the first image features of the image and determines the recommended classification result of the image based on the first image features; then, the Siamese classification network 12 determines the matching degree between the image and the target standard image based on the recommended classification result, and outputs the hierarchical classification result of the image based on the matching degree; wherein, when the matching degree is greater than a threshold, the recommended classification result is output, and the target standard image is the standard image corresponding to the recommended classification result.

[0121] like Figure 5 As shown in the figure, this embodiment of the invention provides a method for classifying image levels, which may further include the following steps:

[0122] 501. Obtain multiple training data pairs.

[0123] In this embodiment of the invention, the classification device can pre-train the classification model. During the training process, multiple training data pairs can be acquired, each training data pair including a training image and the corresponding training classification result.

[0124] 502. Train the model based on multiple training data to obtain an image hierarchical classification model.

[0125] In this embodiment of the invention, the classification device can learn the relationship between each training image and the corresponding training classification result, thereby obtaining the correspondence between the image and the classification, that is, the classification model of the image can be obtained. In this way, when the classification device inputs the image of the object to be tested into the classification model of the image, it can directly obtain the classification result of the image through the classification model.

[0126] 503. Obtain the image of the object to be tested.

[0127] 504. Input the image into the image classification model to obtain the image classification result.

[0128] In this embodiment of the invention, the description of steps 503 to 504 is the same as the detailed description of steps 401 to 402 in the above embodiments, and will not be repeated in this embodiment of the invention.

[0129] like Figure 6 As shown in the figure, this embodiment of the invention provides a method for classifying image levels, which may further include the following steps:

[0130] 601. Obtain multiple training data pairs.

[0131] 602. Recommend training network based on multiple training data.

[0132] In this embodiment of the invention, since the hierarchical classification model includes a recommended classification network and a twin classification network, the hierarchical classification device can first train the recommended training network based on multiple training data.

[0133] 603. When the recommendation training network is in a convergent state, the recommendation classification network is obtained.

[0134] In this embodiment of the invention, when the recommended training network is in a convergent state, it can be said that the recommended training network has been trained. At this time, the ranking classification device can freeze the weights of the recommended training network to obtain the recommended classification network and reconnect the output head of the twin training network.

[0135] 604. Train the Siamese training network based on multiple training data pairs and the recommendation classification network.

[0136] In this embodiment of the invention, after the recommendation classification network has been trained, the ranking classification device can train the Siamese training network based on multiple training data pairs and the recommendation classification network.

[0137] 605. When the twin training network is in a convergent state, the twin classification network is obtained.

[0138] In this embodiment of the invention, when the twin training network is in a convergent state, it can be said that the twin training network has been trained. At this time, the ranking classification device can freeze the weights of the twin training network to obtain the twin classification network. Then, the recommendation classification network and the twin classification network that have been trained can be combined to form a ranking classification model.

[0139] 606. Obtain the image of the object to be tested.

[0140] 607. Input the image into the image classification model to obtain the image classification result.

[0141] In this embodiment of the invention, the description of steps 606 to 607 is the same as the detailed description of steps 401 to 402 in the above embodiments, and will not be repeated in this embodiment of the invention.

[0142] like Figure 7As shown, an embodiment of the present invention provides an image grading device, the grading device comprising:

[0143] The acquisition module 701 is used to acquire an image of the object to be tested;

[0144] The grading module 702 is used to input images into the image grading model to obtain the image grading results;

[0145] The hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network is used to extract the first image features of the image and determine the recommendation classification result of the image based on the first image features. The Siamese classification network is used to determine the matching degree between the image and the target standard image based on the recommendation classification result and output the hierarchical classification result of the image based on the matching degree. When the matching degree is greater than a threshold, the recommendation classification result is output, and the target standard image is the standard image corresponding to the recommendation classification result.

[0146] Optional, such as Figure 8 As shown, the classification device also includes:

[0147] The model training module 703 is used to acquire multiple training data pairs, each training data pair including a training image and the training level classification result corresponding to the training image;

[0148] The model training module 703 is also used to train the model based on multiple training data sets to obtain a hierarchical classification model for images.

[0149] Optionally, the model training module 703 is specifically used to train a network based on multiple training data for training recommendations.

[0150] The model training module 703 is specifically used to obtain the recommendation classification network when the recommendation training network is in a convergent state.

[0151] The model training module 703 is specifically used to train the Siamese training network based on multiple training data pairs and the recommendation classification network.

[0152] The model training module 703 is specifically used to obtain the twin classification network when the twin training network is in a convergent state.

[0153] In this embodiment of the invention, each module can implement the image hierarchical classification method provided in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0154] like Figure 9 As shown, embodiments of the present invention also provide an electronic device, which may include:

[0155] Memory 901 storing executable program code;

[0156] Processor 902 coupled to memory 901;

[0157] Specifically, the processor 902 calls the executable program code stored in the memory 901 to execute the image classification method executed by the image classification device in the above method embodiments.

[0158] This invention provides a computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps of the methods described in the above embodiments.

[0159] This invention also provides a computer program product, wherein when the computer program product is run on a computer, the computer performs some or all of the steps of the methods described in the above method embodiments.

[0160] This invention also provides an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer performs some or all of the steps of the methods described in the above method embodiments.

[0161] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to the invention.

[0162] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0163] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0166] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

Claims

1. A hierarchical classification model for images, characterized in that, The hierarchical classification model includes a recommendation classification network and a twin classification network, wherein the recommendation classification network and the twin classification network are connected. The recommendation classification network is used to extract first image features from the image and determine the recommendation classification result of the image based on the first image features; the image is an appearance image of the object to be tested. The twin classification network is used to determine the matching degree between the image and the target standard image based on the recommended classification result, and output the level classification result of the image based on the matching degree; wherein when the matching degree is greater than a threshold, the recommended classification result is output, and the target standard image is the standard image corresponding to the recommended classification result; when the matching degree is less than the threshold, an abnormal prompt message is output, and the abnormal prompt message is used to indicate that there is an abnormality in the level classification of the image; the level classification result characterizes the appearance level corresponding to the object under test, and the level classification result is used to determine the product quality of the object under test; The number of recommended classification results is at least two; the Siamese classification network includes a feature comparison module. The hierarchical classification model also includes a probabilistic flexible maximum output module, which is connected to both the first output module and the feature comparison module. The probability flexibility maximum output module is used to sort multiple recommendation classification results according to a preset order and output at least two sorted recommendation classification results to the feature comparison module. The feature comparison module is used to obtain the target standard image based on one of the sorted recommended classification results, and to compare the first image feature of the image with the second image feature of the target standard image to obtain the comparison result, wherein one of the sorted recommended classification results is selected according to the preset order.

2. The hierarchical classification model according to claim 1, characterized in that, The recommendation classification network includes a feature extraction module, a feature classification module, and a first output module. The feature extraction module is connected to the feature classification module, and the feature classification module is connected to the first output module. The feature extraction module is used to extract the first image features from the image; The feature classification module is used to classify the first image features using a region feature classification method to obtain the feature classification result. The first output module is used to output the recommended classification result based on the feature classification result.

3. The hierarchical classification model according to claim 2, characterized in that, The Siamese classification network further includes a feature matching module and a second output module. The feature comparison module is connected to the feature matching module, and the feature matching module is connected to the second output module. The feature comparison module is used to obtain the target standard image based on the recommended classification result, and compare the first image feature of the image with the second image feature of the target standard image to obtain a comparison result; The feature matching module is used to determine the matching degree based on the comparison result; The second output module is used to output the level classification result of the image based on the matching degree.

4. The hierarchical classification model according to claim 3, characterized in that, The second output module is configured to output the recommended classification result when the matching degree is greater than or equal to the threshold; and to output an error message when the matching degree is less than the threshold, the error message indicating that there is an error in the level classification of the image.

5. The hierarchical classification model according to claim 3 or 4, characterized in that, The hierarchical classification model also includes a feature sharing module, which is connected to both the feature extraction module and the feature comparison module. The feature sharing module is used to obtain the first image features from the feature extraction module and send the first image features to the feature comparison module.

6. The hierarchical classification model according to claim 1, characterized in that, The hierarchical classification model also includes an iteration count determination module, which is connected to both the second output module and the feature comparison module. The iteration count determination module is used to record the number of times the matching degree is less than the threshold. When the number is less than or equal to a preset number, it instructs the feature comparison module to obtain the updated target standard image in the preset order, so that the second output module outputs the image's level classification result based on the updated target standard image. The updated target standard image is determined based on other sorted recommended classification results besides one of the sorted recommended classification results.

7. A method for classifying images by hierarchy, characterized in that, The method includes: Acquire an image of the object to be tested; the image is an appearance image of the object to be tested. The image is input into an image grading model to obtain the image grading result; the grading result represents the appearance grade of the object under test, and the grading result is used to determine the product quality of the object under test; The hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network extracts a first image feature of the image and determines a recommendation classification result for the image based on the first image feature. The Siamese classification network determines the matching degree between the image and a target standard image based on the recommendation classification result and outputs the hierarchical classification result of the image based on the matching degree. When the matching degree is greater than a threshold, the recommendation classification result is output; when the matching degree is less than the threshold, an error message is output, indicating that the hierarchical classification of the image is abnormal. The target standard image is a standard image corresponding to the recommendation classification result, and the hierarchical classification model is the hierarchical classification model as described in any one of claims 1-6.

8. The method according to claim 7, characterized in that, Before inputting the image into the image hierarchical classification model to obtain the image hierarchical classification result, the method further includes: Acquire multiple training data pairs, each training data pair including a training image and the training level classification result corresponding to the training image; The model is trained based on the multiple training data sets to obtain the hierarchical classification model of the image.

9. The method according to claim 8, characterized in that, The step of training the model based on the multiple training data sets to obtain the hierarchical classification model of the image includes: Based on the multiple training data, a training network is recommended for training. When the recommendation training network is in a convergent state, the recommendation classification network is obtained; The Siamese training network is trained based on the multiple training data pairs and the recommendation classification network. When the twin training network is in a convergent state, the twin classification network is obtained.

10. An image classification device, characterized in that, include: The acquisition module is used to acquire an image of the object to be tested; the image is an appearance image of the object to be tested. The grading module is used to input the image into the image grading model to obtain the grading result of the image; The classification result represents the appearance grade of the object under test, and the classification result is used to determine the product quality of the object under test. The hierarchical classification model includes a recommendation classification network and a Siamese classification network, which are connected. The recommendation classification network extracts a first image feature of the image and determines a recommendation classification result for the image based on the first image feature. The Siamese classification network determines the matching degree between the image and a target standard image based on the recommendation classification result and outputs the hierarchical classification result of the image based on the matching degree. When the matching degree is greater than a threshold, the recommendation classification result is output; when the matching degree is less than the threshold, an error message is output, indicating that the hierarchical classification of the image is abnormal. The target standard image is a standard image corresponding to the recommendation classification result, and the hierarchical classification model is the hierarchical classification model as described in any one of claims 1-6.

11. An electronic device, characterized in that, include: Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the image grading method as described in any one of claims 7 to 9.

12. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the image grading method as described in any one of claims 7 to 9.

Images