Image classification method and computer equipment

Abstract

The invention discloses an image classification method and computer equipment, and relates to the field of image processing, after a decision tree classifier is constructed according to known abnormal priori knowledge in an industrial detection scene, a to-be-classified image in the industrial detection scene is acquired, feature extraction is performed on the to-be-classified image, and the feature of the to-be-classified image is extracted; after abnormal features corresponding to all abnormal attributes in images to be classified are obtained, the abnormal features are input into the decision-making tree classifier to determine a target image set containing all the abnormal features in the decision-making tree classifier, and after the matching degree between the abnormal features and target features of the same abnormal attribute in the target image set is calculated, the abnormal attributes of the images to be classified are obtained. And determining a target anomaly category of the to-be-classified image, such as an unknown anomaly or a known anomaly to which the target image set belongs.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and particularly relates to an image classification method and a computer device. Background Art

[0002] With the continuous progress and development of artificial intelligence technology, image classification technology has been widely applied to many fields such as industry, security, medical care, transportation, or retail. Among them, in the industrial inspection scenario, a classifier can be constructed based on rules to automatically achieve product surface defect detection, appearance classification, component identification, assembly quality inspection, and equipment fault diagnosis, etc., so as to improve production efficiency, quality control level, and equipment operation stability.

[0003] However, in the face of classification scenarios with numerous categories and complex visual features, the currently rule-based constructed classifier cannot be compatible with the discriminant conditions of all categories, and each workflow in the manually constructed classifier is difficult to effectively fit the real discriminant function, resulting in a very low classification accuracy for the input image, and it is impossible to judge unknown categories, and manual detection is still required, which affects the industrial inspection efficiency and accuracy. Summary of the Invention

[0004] In view of the above problems, the present application provides an image classification method and a computer device, and the specific solutions are as follows:

[0005] The first aspect of the present application provides an image classification method, and the method includes:

[0006] Obtain an image to be classified in an industrial inspection scenario;

[0007] Extract features from the image to be classified to obtain abnormal features corresponding to each abnormal attribute in the image to be classified;

[0008] Input the abnormal features into the constructed decision tree classifier. After determining the target image set containing each of the abnormal features in the decision tree classifier, determine the target abnormal category of the image to be classified by calculating the matching degree between the abnormal features and the target features of the same abnormal attribute in the target image set;

[0009] Wherein, the decision tree classifier is constructed based on the prior knowledge of each known abnormality in the industrial inspection scenario, and the abnormal attribute is an attribute that the known abnormality has determined according to the prior knowledge;

[0010] The target abnormal category is an unknown abnormality or a known abnormality to which the target image set belongs.

[0011] In an alternative implementation, determining the target abnormal category of the image to be classified by calculating the matching degree between the abnormal feature and the target feature of the same abnormal attribute in the target image set includes:

[0012] Determine the target features of different abnormal attributes of each target image in the target image set;

[0013] Calculate the matching degree between the abnormal feature and the target feature corresponding to the same abnormal attribute, and determine the matching degree between the image to be classified and the target image set in this abnormal attribute;

[0014] Determine the target abnormal category of the image to be classified according to the matching degree;

[0015] Wherein, the higher the matching degree indicates the higher the probability that the image to be classified and the target image set belong to the same abnormal category;

[0016] If the image to be classified and the target image set do not belong to the same abnormal category, the target abnormal category is the unknown abnormality.

[0017] In an alternative implementation, determining the target abnormal category of the image to be classified according to the matching degree includes:

[0018] Determine the posterior weight of the corresponding abnormal attribute according to the classification importance degree of each abnormal attribute represented by the decision tree classifier;

[0019] Determine the probability that the image to be classified and the target image set belong to the same abnormal category according to the matching degree corresponding to each abnormal attribute and the posterior weight;

[0020] Determine the target abnormal category of the image to be classified according to the probability.

[0021] In an alternative implementation, calculating the matching degree between the abnormal feature and the target feature of the same abnormal attribute in the target image set includes at least one of the following:

[0022] Determine the matching degree between the image to be classified and the target image set in abnormal color according to the abnormal pixel values included in the abnormal feature, and the pixel expected value and pixel standard deviation of each target pixel value of the known abnormality in the target image set;

[0023] Determine the matching degree between the image to be classified and the target image set in abnormal position according to the abnormal position information included in the abnormal feature, and the target position distribution of the known abnormality in the target image set;

[0024] Determine the matching degree of the image to be classified and the target image set in terms of abnormal dimensions based on the abnormal dimension values included in the abnormal features, and the dimension expected values and dimension standard deviations of the target dimension values of the known abnormalities in the target image set;

[0025] Determine the matching degree of the image to be classified and the target image set in terms of abnormal edge information based on the abnormal edge information features included in the abnormal features, and the target edge information features of the known abnormalities in the target image set;

[0026] Determine the matching degree of the image to be classified and the target image set in terms of abnormal high-dimensional features based on the abnormal high-dimensional feature values included in the abnormal features, and the target high-dimensional feature values of the known abnormalities in the target image set.

[0027] In an alternative implementation, constructing the decision tree classifier based on the prior knowledge of each known abnormality in the industrial detection scenario includes:

[0028] Obtain the total image set in the industrial detection scenario; the total image set contains multiple images with different known abnormalities;

[0029] Based on the prior knowledge of each known abnormality in the industrial detection scenario, determine the abnormal attributes for identifying the known abnormalities, and the prior weights corresponding to each of the abnormal attributes; the prior weights indicate the importance of the corresponding abnormal attributes for identifying the known abnormality to which the image belongs;

[0030] Recursively split the total image set that constitutes the initial decision tree according to the known abnormalities, the abnormal attributes, and the prior weights of the multiple images until the split image sets belong to the same known abnormality, to obtain the decision tree classifier;

[0031] Among them, the total image set and the image sets after each split are determined as the nodes of the decision tree classifier, and the hierarchical relationship between the nodes is determined according to the recursive split order of the total image set.

[0032] In an alternative implementation, each image in the total image set is configured with an abnormal label corresponding to the known abnormality of the image, and the position information of the image area where the known abnormality is located;

[0033] The recursively splitting the total image set that constitutes the initial decision tree according to the known abnormalities, the abnormal attributes, and the prior weights of the multiple images until the split image sets belong to the same known abnormality includes:

[0034] Determine the target features corresponding to each of the abnormal attributes of the corresponding image according to the position information corresponding to each of the multiple images;

[0035] Determine the information gain rate of each of the abnormal attributes according to the abnormal labels corresponding to each of the multiple images and the target features of each of the abnormal attributes;

[0036] Recursively split the total image set according to the information gain rate and the prior weight until the split image sets belong to the same known abnormality.

[0037] In an alternative implementation, determining the prior weight corresponding to each abnormal attribute according to the prior knowledge of each known abnormality in the industrial inspection scenario includes any one of the following:

[0038] Respond to the weight assignment operation for each abnormal attribute of the known abnormalities in the industrial inspection scenario, and determine the prior weight of the corresponding abnormal attribute;

[0039] Determine the prior weight of the corresponding abnormal attribute according to the standard deviation of the target features of each known abnormality of the total image set on the same abnormal attribute; the larger the standard deviation of the abnormal attribute, the smaller the corresponding prior weight.

[0040] In an alternative implementation, determining the matching degree of the image to be classified and the target image set in terms of abnormal positions according to the abnormal position information included in the abnormal features and the target position distribution of the known abnormalities in the target image set includes:

[0041] Respectively, according to the abnormal position information included in the abnormal features and the target position information of the known abnormalities in the target image set, obtain the abnormal position mask map of the image to be classified and the target position mask map of the known abnormalities in the corresponding target image;

[0042] Fuse the target position mask maps corresponding to the target image set to obtain the target position density map of the known abnormalities in the target image set;

[0043] Determine the matching degree of the image to be classified and the target image set in terms of abnormal positions according to the target position density map and the abnormal position mask map;

[0044] Wherein, in each position mask map, the mask value belonging to the abnormal position is the first numerical value, and the mask value not belonging to the abnormal position is the second numerical value.

[0045] In an alternative implementation, determining the matching degree of the image to be classified and the target image set in terms of abnormal color based on the abnormal pixel values included in the abnormal feature, and the expected value and standard deviation of each target pixel value with known abnormality in the target image set, includes:

[0046] Obtain the pixel expected value and pixel standard deviation of each target pixel value with known abnormality in the target image set;

[0047] Based on the pixel expected value and the abnormal pixel values included in the abnormal feature, determine the pixel difference between the image to be classified and the target image set in terms of abnormal color;

[0048] Based on the pixel standard deviation and the pixel difference, determine the matching degree of the image to be classified and the target image set in the abnormal color.

[0049] A second aspect of the present application provides a computer device, which includes: at least one communication component, at least one memory, and at least one processor, where:

[0050] The communication component is used to obtain the image to be classified in the industrial detection scenario;

[0051] The memory is used to store multiple computer instructions;

[0052] The processor is used to load and execute the computer instructions to implement the following steps:

[0053] Extract features from the image to be classified to obtain the abnormal features corresponding to each abnormal attribute in the image to be classified;

[0054] Input the abnormal features into the constructed decision tree classifier. After determining the target image set in the decision tree classifier that contains each of the abnormal features, determine the target abnormal category of the image to be classified by calculating the matching degree between the abnormal features and the target features of the same abnormal attribute in the target image set;

[0055] Among them, the decision tree classifier is constructed based on the prior knowledge of each known abnormality in the industrial detection scenario, and the abnormal attribute is the attribute that the known abnormality has determined according to the prior knowledge;

[0056] The target abnormal category is an unknown abnormality or a known abnormality to which the target image set belongs. Description of the Drawings

[0057] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale.

[0058] Figure 1 It is a schematic diagram of a system architecture;

[0059] Figure 2 It is a schematic flowchart of an image classification method provided in the first embodiment of the present application;

[0060] Figure 3 It is a schematic flowchart of an image classification method provided in the second embodiment of the present application;

[0061] Figure 4 It is a schematic flowchart of an image classification method provided in the third embodiment of the present application;

[0062] Figure 5 It is a schematic flowchart of an image classification method provided in the fourth embodiment of the present application;

[0063] Figure 6 It is a schematic flowchart of an image classification method provided in the fifth embodiment of the present application;

[0064] Figure 7 It is a schematic structural diagram of an image classification device provided in the embodiments of the present application;

[0065] Figure 8 It is a schematic diagram of an optional hardware structure of a computer device applicable to the image classification method proposed in the embodiments of the present application;

[0066] Figure 9 It is a schematic diagram of another optional hardware structure of a computer device applicable to the image classification method proposed in the embodiments of the present application. Specific Embodiments

[0067] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only for explaining the specific embodiments of the present application and are not intended to limit the present application. The embodiments of the present application will be described below with reference to the drawings. Those skilled in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0068] In the context of the present application and in the above-mentioned drawings, terms such as "first", "second", etc. are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0069] The present application can be applied to industrial inspection application scenarios to classify and identify images of objects to be detected, so as to achieve product defect detection, assembly quality detection, equipment fault diagnosis, etc. Taking the implementation of the product defect classification task in the industrial inspection scenario as an example, multiple application scenarios implemented in products will be introduced below. In practical applications, the present application can be but is not limited to being applied to terminal devices or servers with data processing capabilities, and may also be realized based on the communication between terminal devices and servers, which can be determined according to actual needs, and the present application does not limit this.

[0070] Refer to Figure 1 , Figure 1 which shows a schematic diagram of a system architecture. The system may include a terminal 110 and a server 120, where:

[0071] According to the above analysis, the terminal 110 may complete the action of obtaining a processing result (such as the target abnormal category of the image to be classified, etc.) based on the received parameters (such as the processing instruction for the image to be classified displayed on the detection interface) by itself, without the cooperation of the server, and implement the image classification method provided by the embodiments of the present application.

[0072] In some alternative implementations, the server 120 may also provide the image classification method provided by the embodiments of the present application for one or more terminals 110. In this case, the terminal 110 may send the received parameters (such as the image to be classified collected in the industrial inspection scenario and its abnormal classification request / instruction, etc.) to the server 120. After the server 120 obtains the processing result (such as the target abnormal category of the image to be classified and its abnormal features, etc.) based on the received parameters, it returns the processing result to the terminal 110.

[0073] In practical applications, the terminal 110 in the above system may include, but is not limited to: a mobile phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, a robot in an industrial inspection scenario, or an independent image acquisition device (such as a camera or a video recorder), or other industrial terminals, etc. One or more of them can determine the product form of the terminal 110 according to the main device that executes the method provided in the embodiments of the present application. The present application does not limit this.

[0074] In different system scenarios for implementing the method provided in the embodiments of the present application, the terminal 110 may include all or some of the components such as a radio frequency unit, a processor, a memory, an input unit, a display unit, a camera, an audio circuit, a speaker, a pick-up (such as a microphone), an external interface, and a power supply, etc. It should be understood that these components described in the present application are only examples and do not constitute a limitation on the terminal or the multifunctional device. According to actual processing requirements, the terminal 110 may include more or fewer components, or combine certain components, or different components.

[0075] Among them, the input unit in the terminal 110 can be used to receive input data, which can be used to generate user setting parameters for certain functional devices, or signal inputs related to function control, or input information such as numbers / characters, etc. In practical applications, the input unit may include, but is not limited to: a touch screen, a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, a pick-up, and an image collector, etc. One or more of them can complete data input according to the input method required for various input data and use the input unit that supports this input method. The implementation process is not elaborated with examples in the present application.

[0076] The display unit in the terminal 110 can be used to display the data input by the user or the data provided to the user, various interactive menus / interfaces of the terminal 110, or the playback of image / video files (such as monitoring video files in an industrial inspection scenario, etc.). In the embodiments of the present application, the display unit can be used to display an image detection interface to display the image to be classified and its classification results on this image detection interface, such as the target abnormal category and the abnormal features corresponding to different abnormal attributes, such as the abnormal position information marked by a detection frame, etc.

[0077] The memory in the terminal 110 can be used to store software codes / computer instructions related to implementing the image classification method proposed in the present application. The processor can execute the software codes / computer instructions to implement the image classification method proposed in the present application, or can also schedule other units (such as the above input unit and display unit, etc.) to implement corresponding functions.

[0078] In addition, the external interface in the terminal 110 can be used to implement communication between the terminal 110 and other devices, such as receiving images in an industrial inspection scenario collected in real time by other independent image acquisition devices, or can also be used to connect to a charging device to charge the terminal 110, etc. The functions and implementations of other components of the terminal 110 can be determined according to actual scenario requirements, and this application does not give detailed examples one by one here.

[0079] In some embodiments, in a system scenario as Figure 1 shown, that is, the server 120 cooperates with the terminal 110 to implement the method provided in the embodiments of this application. One or more terminals 110 can be deployed for image acquisition for each detection object / detection position, etc. in the industrial inspection scenario. After that, the radio frequency unit of the terminal 110 can send the collected images to be classified to the server 120 for processing and receive the processing results returned by the server 120. It should be understood that the images to be classified can also be preprocessed images with relatively high clarity automatically screened by the terminal 110 or manually screened by the user. After that, in addition to using the radio frequency unit to implement communication with the server 120, communication between the terminal 110 and the server 120 can also be implemented through other communication modules, such as a wireless network port, etc. This application does not elaborate on the communication method and its communication principle between the two.

[0080] Based on this, the server 120 can include a communication bus, communication components, a memory, and a processor. Communication can be carried out between the communication components, the memory, and the processor through the communication bus. The functions of the memory and the processor are similar to those of the memory and the processor of the terminal 110, so that the server 120 can implement the method provided in the embodiments of this application, and the implementation process is not described in detail in this embodiment.

[0081] Combined with the above analysis, to solve the problems proposed in the background art part, the embodiments of this application provide an image classification method. The image classification method of the embodiments of this application will be introduced in detail below with reference to the accompanying drawings.

[0082] Referring to Figure 2 , which is a schematic flowchart of an image classification method provided in Embodiment 1 of this application, and can be applied to a computer device. Combined with the above analysis, the computer device can be the above terminal or server. As shown in FIG. 2, the image classification method proposed in this embodiment can include but is not limited to the following steps:

[0083] Step S21, obtaining an image to be classified in an industrial inspection scenario;

[0084] In the embodiments of the present application, the image to be classified may be obtained by collecting images of the object to be detected (such as industrial products) in an industrial detection scenario, such as the original image of the current frame collected in real time, or after preprocessing a plurality of frames of images continuously collected for the same object to be detected, selecting a frame of image that meets the image input requirements of the subsequent classification task from the preprocessed images as the image to be processed, and so on.

[0085] Among them, the preprocessing process of the collected image is used to improve the image quality and provide a standardized input for the subsequent classification task. The preprocessing may include, but is not limited to: image denoising processing to reduce the interference of noise data on abnormal classification; image color space conversion processing to obtain a unified color space format, such as the RGB format, etc.; image size normalization to obtain the image size required for subsequent processing; one or a combination of geometric transformations such as rotation / flip / correction, etc., which can be determined according to actual needs and implemented using corresponding processing tools. The present application does not elaborate on the implementation methods of each image preprocessing.

[0086] In some alternative implementations, the image to be classified may also be any one frame of image or any one frame of image selected from multiple frames of images selected by relevant personnel from the collected multiple frames of original images or preprocessed images using the input unit of the computer device according to the image input requirements of the subsequent classification task. In this case, the computer device may determine the image to be classified in response to an image selection instruction in the industrial detection scenario. Regarding the type of the input unit and how to use the input unit to select the image to be classified, the implementation process may refer to the description of the corresponding part of the above embodiments, and this embodiment will not elaborate here.

[0087] Step S22: Extract features from the image to be classified to obtain abnormal features corresponding to each abnormal attribute in the image to be classified; the abnormal attribute is an attribute possessed by a known abnormality determined based on the prior knowledge of each known abnormality in the industrial detection scenario.

[0088] Following the above analysis, in order to determine the abnormal category to which the abnormality existing in the image to be classified belongs, especially an unknown abnormality in the industrial detection scenario, that is, an abnormal category different from each known abnormality existing in the industrial detection scenario, so as to implement a classification task or other downstream tasks accordingly, etc., it is possible to determine which attributes of various known abnormalities existing in the industrial detection scenario play an important role in identifying the known abnormality based on the prior knowledge of the industrial detection scenario, and these attributes can be regarded as abnormal attributes. That is to say, based on the characteristics of each abnormal attribute in each image with known abnormalities in the work detection scenario, it is possible to reliably and accurately determine which known abnormality the image belongs to. The present application does not limit the content of each abnormal attribute.

[0089] In this application, prior knowledge refers to the known information about defects, faults, or other abnormal patterns that appear in industrial inspection scenarios, including but not limited to their morphology, location, frequency, and causes, etc., which is used to significantly improve the efficiency and accuracy of anomaly detection. Especially in the case of sparse data (few images with known anomalies) or complex backgrounds in industrial inspection scenarios, the prior knowledge of known anomalies can be recorded or statistically analyzed during the industrial inspection process. This application does not limit the content and acquisition method of the prior knowledge of known anomalies in industrial inspection scenarios.

[0090] Among them, the abnormal morphologies included in the prior knowledge of industrial inspection scenarios can include: known abnormal morphologies such as scratches (linear), holes (circular), cracks (fractal structures), and stains (irregular patches); defects usually have a fixed size range (such as the diameter of welding bubbles is less than 1 mm); there are also abnormal texture changes such as local texture breaks (such as fabric holes) and direction mutations (such as inconsistent rolling patterns on the metal surface). The location in the prior knowledge can be the abnormal spatial distribution including location preference and directionality. This location preference can include abnormal high-occurrence areas such as welding defects concentrated at the weld, edge chipping prone to occur at the edge, and wear prone to occur at the assembly joint surface; multiple anomalies may appear in clusters (such as continuous spots caused by uneven spraying), etc., spatial correlations. The abnormal location directionality can be related to processing marks, such as the defects of turned parts are distributed along the axial direction, and the defects of stamped parts are related to the movement direction of the die, etc.

[0091] In addition, the causes included in the above prior knowledge usually refer to the physical causes of anomalies, usually process-related defects / anomalies, and anomalies caused by environmental interferences (such as oil stains, dust, and light reflections that are easily misjudged as defect interference sources). The former can include product defects caused by abnormal processing parameters, such as burning caused by too high temperature and false soldering caused by insufficient pressure, or anomalies caused by industrial material defects, such as circular shadows of internal air holes in metal under X-rays and sunken plastic injection shrink marks. The frequency of the above prior knowledge can be expressed as time-series dynamic information, such as periodic anomalies such as defects caused by mold wear increasing over time and periodic scratches caused by equipment vibration, and movement target anomalies such as position offset or rotation out-of-tolerance of parts on the conveyor belt. It should be noted that the content of the above prior knowledge is not limited to the examples given in this application.

[0092] Based on the above description of prior knowledge, it is possible to analyze the quantitative description and classification features of known anomalies in the industrial inspection scenario, and determine the categories of each anomaly attribute of each known anomaly, including but not limited to geometric attributes such as the position, size, and shape of the known anomalies listed above, and may also include apparent attributes such as color, texture, and reflection characteristics; as well as physical attributes such as depth / height and material characteristics; and dynamic attributes such as temporal changes and motion characteristics. Since this application uses image classification to determine the anomaly category, attributes that cannot be reflected in a single-frame image can be not analyzed, such as physical attributes and dynamic attributes. Optionally, if this application can analyze consecutive multi-frame images to determine the anomaly category, it can also be achieved by combining dynamic attributes. Of course, the anomaly characteristics that may occur in the objects to be detected in different industrial inspection scenarios may vary, and the anomaly attributes determined based on their prior knowledge may be inconsistent or may be the same, depending on the situation.

[0093] According to but not limited to the method described above, after determining each anomaly attribute of the known anomalies in the industrial inspection scenario, before detecting the anomaly category of the currently obtained image to be classified, feature extraction can be performed on the image to be classified to obtain the anomaly features corresponding to each anomaly attribute in the image to be classified. In this implementation process, the image to be classified can be input into a pre-trained feature extraction network / object recognition model or a dedicated anomaly detection network to obtain the anomaly features corresponding to each of the multiple anomaly attributes in the image to be classified.

[0094] In an alternative implementation, this application can also select appropriate feature extraction methods according to the feature characteristics of different anomaly attributes, and perform feature extraction on the anomaly regions in the image to be classified for the corresponding anomaly attributes respectively to obtain the anomaly features of the corresponding anomaly attributes. For example, for the anomaly size, after determining the anomaly region, the total number of pixels included in the anomaly region or the number of pixels of the length and width respectively can be identified and determined as the size of the anomaly in the image to be classified. For the anomaly color feature, it can be obtained based on methods such as color histogram or color difference calculation; for texture features such as edges, corresponding texture detection algorithms can be used; for high-dimensional features such as anomaly semantics, feature extraction can be performed on the image to be classified based on deep learning algorithms. For the implementation method of step S22, including but not limited to the several implementation methods listed in this application, it can be configured or adjusted according to actual needs to ensure that the various anomaly features extracted are reliable and accurate, laying a reliable foundation for the accurate detection of the subsequent anomaly category.

[0095] Step S23, input the anomaly features into the constructed decision tree classifier. After determining the target image set containing each anomaly feature in the decision tree classifier, determine the target anomaly category of the image to be classified by calculating the matching degree between the anomaly features and the target features of the same anomaly attribute in the target image set.

[0096] In practical applications, since the classifier constructed based on rules is a classifier with a linear structure, it usually determines the abnormal category of the image to be classified by comparing the abnormal attribute thresholds corresponding to different pre-configured abnormal categories with the actual abnormal attribute values of the image to be classified. It is difficult to take into account the judgment conditions of all abnormal categories, especially in the classification scenario with numerous abnormal categories and complex visual features, which reduces the accuracy and reliability of image classification.

[0097] To solve the above problems, this application proposes to use a tree structure instead of a linear structure to construct a classifier. And to improve the efficiency and interpretability of image classification and support the deployment of the classifier on computer devices with limited hardware resources (such as industrial control computers or edge devices) to achieve real-time abnormal category detection of the collected images, etc., this application will construct a classifier with a tree structure of a decision tree, that is, a decision tree classifier. In this construction process, each abnormal attribute of the known abnormalities in the industrial detection scenario can be used as the node attribute of each node in the decision tree. When splitting each node into multiple child nodes, combined with the prior knowledge of each known abnormality in the industrial detection scenario, the most appropriate abnormal attribute is selected as the splitting attribute to split the node according to this splitting attribute, forming the intermediate nodes and leaf nodes of the decision tree. It can be seen that in the face of a classification scenario with numerous categories and complex visual features, compared with the linear structure, the decision tree classifier constructed in this application, which is a multi-level classifier, can flexibly construct and combine multiple strategies (i.e., the node splitting strategy based on the selected splitting attribute) to classify complex local visual features, improving the classification accuracy.

[0098] Among them, the nodes in the decision tree classifier represent the image sets of known abnormalities in the industrial detection scenario. Each image in this image set is configured with the target features corresponding to each abnormal attribute, and the features of each abnormal attribute possessed by the corresponding known abnormality. Since the nodes of the decision tree classifier have a tree structure, its construction process starts from the top node of the tree structure, that is, the root node, which is used as the node to be split. The image set is divided according to the splitting attribute of each image in the image set it represents, and each subset after division is used as the child node of the node to be split. Then, each child node is used as a new node to be split, and the image set division step is repeated to gradually create the intermediate nodes of each layer of the decision tree until each image included in the subset after division belongs to the same category of known abnormality, and this subset constitutes the leaf node of the decision tree, thus completing the construction of the decision tree.

[0099] It can be seen that each intermediate node and leaf node in the decision tree classifier constructed in this application are formed by dividing the image set represented by the parent node of the node (intermediate node or leaf node) into subsets. The feature threshold of the splitting attribute on which the division process depends can be determined as the feature matching condition of the corresponding subset representing the node, that is, the feature classification condition for dividing the image set represented by the parent node. This application does not limit the content of this condition. It should be understood that since this decision tree classifier is constructed based on the prior knowledge of each known abnormality in the industrial detection scenario, compared with the technical personnel in the industrial detection scenario manually constructing the feature classification condition based on experience, the feature classification condition determined in this application is determined in combination with the prior knowledge of the industrial detection scenario, so that the content of this condition represents the true feature differences of different known abnormalities in the industrial detection scenario, in order to solve the problem of low classification accuracy of the manually constructed classifier.

[0100] Based on this, in the embodiment of this application, after constructing a decision tree classifier according to the method described above based on the prior knowledge of each known abnormality in the industrial detection scenario, the abnormal features corresponding to each abnormal attribute in the image to be classified can be input into this decision tree classifier, and feature matching is performed layer by layer from the root node of the decision tree classifier to the child nodes. For example, according to the feature classification condition corresponding to the node, it is determined which image set represented by the node the abnormal features of the image to be classified belong to, that is, the feature ranges of the multiple child nodes to be matched (that is, the feature thresholds corresponding to each abnormal attribute of the corresponding child node) are respectively compared with the abnormal features of the same abnormal attribute, and the child node corresponding to the feature range where the abnormal feature is located is determined as the matching node, that is, the node containing the abnormal feature. Then, this matching node is used as the parent node, and the above feature matching step is repeated for its each child node until the matching node determined this time is a leaf node. The leaf node is determined as the target node, and the image set it represents is determined as the target image set containing the abnormal features in the image to be classified.

[0101] At this time, this application determines one of the known abnormalities to which the target image set represented by the above determined target node belongs as the pending abnormal category, rather than directly taking the known abnormality represented by the leaf node as the target abnormal category. This application will further perform a matching calculation on the target features corresponding to each abnormal attribute in the target image set and the abnormal features corresponding to the same abnormal attribute of the image to be classified, and comprehensively determine the matching degree between the target image set and the image to be classified based on the matching degrees of the features corresponding to each abnormal attribute. The higher the matching degree, the greater the probability that the abnormality in the image to be classified belongs to the same abnormal category as one of the known abnormalities to which the target image set belongs. Thus, it can be more accurately determined whether the abnormality in the image to be classified belongs to this known abnormality. If not, it means that the target abnormal category of the image to be classified is an unknown abnormality. If so, it means that the target abnormal category of the image to be classified is one of the known abnormalities to which the target image set belongs.

[0102] In summary, according to the prior knowledge of each known anomaly in the industrial inspection scenario, this application constructs a decision tree classifier with a multi-level tree structure, which is more suitable for classification tasks with a large number of categories and complex visual features compared to rule-based classifiers. After extracting the anomaly features corresponding to each anomaly attribute from the image to be classified, the decision tree classifier is input for feature matching. After determining the target image set containing each anomaly feature (i.e., the image set represented by a leaf node of the decision tree classifier, and each target image included belongs to the same known anomaly), the matching degree between each anomaly feature and the target feature of the same anomaly attribute in the target image set will be further calculated, and based on this, it is verified whether the image to be classified belongs to this known anomaly, thereby accurately determining that the target anomaly category of the image to be classified is this known anomaly or an unknown anomaly, solving the problem of absolute classification of the decision tree (i.e., directly determining the anomaly category represented by the matched leaf node as the target anomaly category), being able to effectively determine unknown anomalies in the industrial inspection scenario, and improving the accuracy and reliability of image classification.

[0103] Refer to Figure 3 , which is a schematic flowchart of an image classification method provided in the second embodiment of this application. In this embodiment, the processing process of the decision tree classifier for the input anomaly features can be refined after inputting the anomaly features of the image to be classified into the constructed decision tree classifier in the above embodiment. Regarding the construction process of the decision tree classifier and the acquisition process of each anomaly feature, reference can be made to the corresponding parts of the context embodiments, and this embodiment will not be elaborated here. Based on this, as Figure 3 shown, after inputting the anomaly features of the image to be classified into the constructed decision tree classifier, the decision tree classifier can but is not limited to perform the following steps:

[0104] Step S31, determine the target image set in the decision tree classifier that contains the anomaly features corresponding to each anomaly attribute in the image to be classified;

[0105] Combined with the above analysis, for the multi-level tree structure of the decision tree classifier, after extracting the anomaly features corresponding to each anomaly attribute from the image to be classified, starting from the root node at the top layer of the decision tree classifier, according to the feature matching conditions (i.e., feature classification conditions) of its each child node, it is determined whether the anomaly feature is within the feature range of the anomaly attribute corresponding to this child node. If so, this child node is used as the parent node, and in this way of feature matching, continue to select the child node with a feature range that contains the anomaly feature from each child node at its next level, and so on, screening nodes layer by layer until the leaf node containing each anomaly feature is determined, and the image set represented by this leaf node is determined as the target image set.

[0106] Exemplarily, in combination with the relevant descriptions of abnormal attributes and their abnormal features in the above embodiments, if the abnormal features corresponding to different abnormal attributes extracted from the image to be classified include: the pixel values of abnormal colors, the coordinate values of abnormal positions, the aspect ratio of the length and width of the region of abnormal size, and the proportion of edge pixels of abnormal edge information, etc., each node in the decision tree classifier represents that each image in the image set is configured with the target features of these types of abnormal attributes respectively, that is, the values of the corresponding abnormal attributes, thereby constituting the feature range of this node on the corresponding abnormal attribute, that is, the feature threshold for distinguishing different nodes at the same level.

[0107] In this way, after inputting the abnormal features such as the pixel values, coordinate values, aspect ratio of the region length and width, and proportion of edge pixels of the abnormal region in the image to be classified into the decision tree classifier, feature classification can be carried out layer by layer from the root node downward, and the node corresponding to the feature range where the abnormal feature is located can be determined at each level. Since the sum of the value ranges corresponding to the same abnormal attribute included in each leaf node at the same level constitutes the value range of this abnormal attribute included in its parent node. By comparing the abnormal features of the same abnormal attribute with the value ranges of each leaf node at this level, the matching node containing the abnormal feature at this level can be determined. The value range of the abnormal color represented by this matching node includes the pixel values of the abnormal region of the image to be classified, the value range of the abnormal position includes the coordinate values of the abnormal region of the image to be classified, the value range of the abnormal size includes the aspect ratio of the length and width of the abnormal region of the image to be classified, and the value range of the abnormal edge information includes the proportion of edge pixels of the abnormal region of the image to be classified, etc. Repeat the above processing until the determined node belongs to the leaf node, and the preliminary abnormal classification of the image to be classified is completed. One known abnormality possessed by the leaf node, that is, the target image set, can be determined as the pending abnormal category.

[0108] It should be noted that the abnormal attributes and their features of the known abnormalities include but are not limited to the content listed above. For example, the feature of abnormal color can also be the saturation mean / brightness mean or variance, etc., the feature of abnormal size can also be the area ratio, the feature of abnormal position can also be the normalized position of the abnormal center coordinate relative to the image size, or the ratio of the closest distance to the image edge, etc. Of course, the abnormal attributes can also include abnormal high-dimensional features, such as the global average pooling output of the last convolutional layer of the convolutional neural network. In combination with the above analysis, it can be flexibly determined according to the prior knowledge of the industrial detection scenario. In addition, it should be understood that with the accumulation of prior knowledge in the industrial detection scenario, the abnormal attributes or the decision tree classifier can be updated according to the method described above to improve the classification accuracy of the decision tree classifier.

[0109] Step S32, determining the target features of different abnormal attributes possessed by each target image in the target image set;

[0110] Step S33: Calculate the matching degree between the abnormal features and the target features corresponding to the same abnormal attribute, and determine the matching degree between the image to be classified and the target image set in terms of this abnormal attribute;

[0111] To accurately determine unknown abnormalities in industrial inspection scenarios, after initially determining the pending abnormal category (a known abnormality) to which the image to be classified belongs according to the above method, that is, after an abnormal category represented by a leaf node, the target features of each abnormal attribute of each target image included in the target image set represented by this leaf node, that is, the target features of each abnormal attribute of this known abnormality, can further match the features of the same abnormal attribute between the image to be classified and the target image set to obtain the matching degree between the two in terms of this abnormal attribute. For example, the matching degrees corresponding to each abnormal attribute such as abnormal color, abnormal position, and abnormal size between the abnormality of the image to be classified and the known abnormality in the target image set are obtained, and thus the category consistency between the abnormality of the image to be classified and the known abnormality in the target image set is analyzed.

[0112] Among them, the higher the above matching degree, the higher the probability that the image to be classified and the target image set belong to the same abnormal category, and the more credible the image classification result of the pending abnormal category determined through step S31 is the target abnormal category. Therefore, this application can determine the target abnormal category of the image to be classified based on the determined matching degrees corresponding to each abnormal attribute, that is, a known abnormality represented by the target image set, or an abnormality that does not belong to this known abnormality. Since this abnormality also does not belong to the known abnormalities or normal (no abnormality) represented by other leaf nodes, it indicates that the abnormality existing in the image to be classified belongs to an unknown abnormality.

[0113] Step S34: Determine the posterior weight of the corresponding abnormal attribute according to the classification importance of each abnormal attribute represented by the decision tree classifier;

[0114] In practical applications, due to the characteristics of different abnormal attributes of known abnormalities in industrial inspection scenarios, the classification importance for determining the known abnormality in the image is different, that is, the changes in the characteristics of different abnormal attributes have different degrees of influence on determining the abnormal category of the image, which can be reflected by the hierarchical structure of the constructed decision tree classifier. To further improve the accuracy of abnormal classification, in an optional implementation method of determining the target abnormal category of the image to be classified based on the matching degrees corresponding to each abnormal attribute, the matching degrees corresponding to each abnormal attribute can be normalized and weighted and summed in combination with the classification importance of different abnormal attributes to determine whether the image to be classified and the target image set belong to the same abnormal category.

[0115] Based on this, the present application can determine the posterior weights of the corresponding abnormal attributes by sorting the abnormal attributes in the multi-level structure of the constructed decision tree classifier, that is, sorting the classification importance degrees of the abnormal attributes, so as to implement the weighted processing of the matching degrees of the different abnormal attributes determined above. Among them, the higher the classification importance degree of the abnormal attribute represented by the decision tree classifier, the higher the posterior weight of the corresponding abnormal attribute, that is, the matching degree corresponding to this abnormal attribute is more important for determining the target abnormal category of the image to be classified.

[0116] In an alternative implementation, the classification importance degrees of the abnormal attributes represented by the decision tree classifier can be mapped to discrete values within [0, 1] to obtain the posterior weights of the corresponding abnormal attributes, so that the sum of all posterior weights is 1. Optionally, the present application can implement step S34 according to a linear function, or pre-determine the corresponding relationship between different classification importance degrees and subsequent weights in advance according to experience or experiments, such as a linear change curve or a piecewise linear function, etc., and accordingly determine the posterior weights corresponding to the classification importance degrees of the abnormal attributes represented by the decision tree classifier. The present application does not limit the specific implementation method of step S34.

[0117] Step S35: Determine the probability that the image to be classified and the target image set belong to the same abnormal category according to the matching degree and the posterior weight corresponding to each abnormal attribute; the higher the matching degree, the higher the probability that the image to be classified and the target image set belong to the same abnormal category;

[0118] Step S36: Determine the target abnormal category of the image to be classified according to this probability; if the image to be classified and the target image set do not belong to the same abnormal category, the target abnormal category is an unknown abnormality.

[0119] Following the above analysis, the posterior weights corresponding to each abnormal attribute can be used to perform weighted summation on the matching degrees corresponding to each abnormal attribute, normalize the total matching degree, and obtain the probability that the image to be classified and the target image set belong to the same abnormal category. Or, after normalizing the matching degrees corresponding to each abnormal attribute, use the posterior weights corresponding to the abnormal attributes to perform weighted summation on the normalized matching degrees corresponding to each abnormal attribute to obtain the probability that the image to be classified and the target image set belong to the same abnormal category.

[0120] Exemplarily, assume that A i represents the i-th abnormal attribute, represents the posterior weight of the i-th abnormal attribute, and M i represents the matching degree of the i-th abnormal attribute between the image to be classified and the target image set. Step S35 can be implemented according to the following formula:

[0121] ; (1)

[0122] In Formula 1, norm() represents a normalization function, which, in combination with the above analysis, can be linear normalization to linearly scale the classification importance of each abnormal attribute to the range of [0, 1]. However, it is not limited to this type of normalization function. |A| represents the number of types (count) of abnormal attributes, and Ω represents the probability that the image to be classified and the target image set belong to the same abnormal category.

[0123] After that, it can be determined whether the above probability is greater than a pre-configured probability threshold. This probability threshold can be determined based on experiments or experience and is the probability critical value for judging whether the image to be classified and the target image set belong to the same abnormal category. For example, it can be the minimum probability that the image to be classified and the target image set belong to the same abnormal category. This application does not limit the size of this probability threshold. In this way, when it is determined that the predicted probability is greater than or equal to the probability threshold, it can be considered that the image to be classified and the target image set belong to the same abnormal category, and the target abnormal category of the image to be classified is determined to be a known abnormal belonging to the target image set. When it is determined that the predicted probability is less than the probability threshold, it can be considered that the image to be classified and the target image set do not belong to the same abnormal category, and the target abnormal category of the image to be classified is determined to be an unknown abnormal.

[0124] It should be noted that in the process of configuring the above probability threshold, it will be determined in combination with the number of abnormal attributes included in the constructed decision tree classifier. During the feature extraction process of the image to be classified, the abnormal features corresponding to the same number of each abnormal attribute will be obtained to ensure that the determined probability is based on the matching degree and posterior weight corresponding to the same number of abnormal attributes. Usually, the more the number of abnormal attributes, the higher the configured probability threshold. This application does not limit the size of the probability threshold under different numbers of abnormal attributes.

[0125] In summary, in the embodiment of this application, during the classification process of the abnormal features corresponding to each abnormal attribute in the image to be classified based on the constructed decision tree classifier, after determining the target image set containing each abnormal feature, the matching degree between each abnormal feature and the target feature of the same abnormal attribute in the target image set will be further calculated. Combining the classification importance of each abnormal attribute represented by the decision tree classifier, the matching degrees of each abnormal attribute are weighted and summed to determine the probability that the image to be classified and the target image set belong to the same abnormal category. Based on this, an effective determination of unknown abnormalities without prior knowledge in the industrial detection scenario is realized, improving the accuracy and reliability of the image classification result, that is, the target abnormal category.

[0126] Refer to Figure 4, which is a schematic flowchart of an image classification method provided in Embodiment 3 of this application. In this embodiment, taking the five abnormal attributes of known abnormalities in the industrial inspection scenario in the above embodiment, including abnormal color, abnormal position, abnormal size, abnormal edge information, and abnormal high-dimensional features, as an example, an optional calculation method for the matching degree between the abnormal features of the image to be classified and the target features of the same abnormal attribute in the target image set is described. However, it is not limited to these five types of abnormal attributes. This application only takes this as an example for illustration. The calculation methods for the matching degrees corresponding to other abnormal attributes are similar, and this application will not give examples one by one. In addition, regarding the extraction process of the abnormal features of each abnormal attribute in the image to be classified, and the implementation process of determining the target abnormal category of the image to be classified based on the matching degrees corresponding to each abnormal attribute, reference can be made to the descriptions of the corresponding parts of the context embodiments, and this embodiment will not elaborate.

[0127] Based on this, as Figure 4 shown, the implementation method for calculating the matching degree between the abnormal features and the target features of the same abnormal attribute in the target image set proposed in this embodiment may include but is not limited to the following steps:

[0128] Step S41, determine the matching degree of the abnormal color between the image to be classified and the target image set according to the abnormal pixel values of the image to be classified, and the pixel expected values and pixel standard deviations of the target pixel values of the known abnormalities in the target image set;

[0129] In the embodiment of this application, when the abnormal attribute includes an abnormal color, the pixel values of the abnormal area can be extracted from the image to be classified and recorded as abnormal pixel values, such as the pixel values on the RGB three channels, etc. At this time, the target pixel values of the known abnormalities corresponding to each target image in the target image set of the decision tree classifier can be obtained, and according to each target pixel value, the pixel expected value and pixel standard deviation of the known abnormality can be calculated. Of course, this application can also pre-calculate the pixel expected values and pixel standard deviations of the known abnormalities corresponding to the image sets represented by each leaf node during the construction of the decision tree classifier. At this time, the pixel expected values and pixel standard deviations of the known abnormalities corresponding to the target image set can be directly read.

[0130] Among them, the pixel expected value represents the average value of the pixel values of the known abnormality in each target image in the target image set, reflecting the average brightness or gray level of the known abnormality. The pixel standard deviation is used to measure the degree of dispersion of the pixel value distribution of the known abnormality in the target image set, reflecting the fluctuation of the known abnormality pixel values. Therefore, this application can analyze the abnormal pixel values of the image to be classified based on this to determine the matching degree of the abnormal color between the image to be classified and the target image set.

[0131] In a possible implementation method, after obtaining the pixel expected value and pixel standard deviation of each target pixel value with known anomalies in the target image set, based on the above analysis of the pixel expected value and pixel standard deviation, the pixel difference in abnormal colors between the image to be classified and the target image set can be determined first according to the pixel expected value and the abnormal pixel value, and then the matching degree in abnormal colors between the image to be classified and the target image set can be determined according to the pixel standard deviation and the pixel difference.

[0132] Based on this, assume that the pixel expected value and pixel standard deviation of the target pixel values with known anomalies in the RGB three channels in the above target image set are calculated according to the following corresponding formulas:

[0133] ; (2)

[0134] ; (3)

[0135] Among them, the target image set is represented as D j , that is, the j-th subset in the total image set D for constructing the decision tree classifier, and j can represent the abnormal category predicted in the decision tree classifier, that is, the above-mentioned undetermined abnormal category; represents the pixel expected value of D j on the RGB three channels; represents the target pixel values of the known anomalies of the i-th target image in the target image set on the RGB three channels, ; represents the pixel standard deviation of D j on the RGB three channels. |D j | can represent the number of target images included in the target image set.

[0136] After that, the matching degree in abnormal colors between the image to be classified and the target image set can be calculated according to formula (4), denoted as M color :

[0137] ; (4)

[0138] In formula (4), represents the pixel values of the abnormal area in the image to be classified on the RGB three channels, that is, the abnormal pixel values included in the above-mentioned extracted abnormal features, (tolerance range), and the value represents a slack variable, which can be determined based on experience or experiments to adjust the relationship of the matching degree of the abnormal color determined in step S41 as an adjustment coefficient; is a minimum floating-point constant to prevent the denominator of formula (4) from being 0, and to avoid the situation where the matching degree of the abnormal color cannot be effectively calculated when the pixel difference in abnormal colors between the image to be classified and the target image set is zero.

[0139] It should be noted that the calculation method for the matching degree of abnormal colors between the image to be classified and the target image set includes, but is not limited to, the calculation method represented by the above formula (4). It can also be determined based on calculating the difference / variance between the abnormal pixel values of the image to be classified and the average value of the target pixels with known abnormalities in the target image set. The implementation process will not be elaborated in this application.

[0140] Step S42: Determine the matching degree of abnormal positions between the image to be classified and the target image set according to the abnormal position information of the image to be classified and the distribution of known abnormal target positions in the target image set.

[0141] Among them, the distribution of known abnormal target positions in the target image set can be determined during the construction of the decision tree classifier, or during the inference process, according to the known abnormal target position information of each target image in the target image set. This application does not limit the calculation method of the distribution of known abnormal target positions in the target image set. This target position distribution represents the spatial characteristics of this type of known abnormality, and can be the density or position range of the image area where the known abnormality is located in the target image set.

[0142] In this way, when extracting the abnormal position information of the abnormal position from the image to be classified, such as the pixel coordinates, region coordinates, or center coordinates of the abnormal region, and its representation method is consistent with the representation method of the target position distribution, the abnormal position information can be compared with the target position distribution to determine whether the abnormal position in the image to be classified is within the position range of the target position distribution, or to determine the probability value that the abnormal position in the image to be classified belongs to the known abnormal position, etc., so as to determine the matching degree of abnormal positions between the image to be classified and the target image set.

[0143] In a possible implementation method, assume that the above target position distribution is represented by a target position density map, that is, the distribution density of a known abnormal region belonging to each target image in the target image set obtained by statistics. Each pixel value therein can represent the number of abnormal pixels at this position, that is, the pixel values of the same known abnormal position in each target image are accumulated. In this case, the abnormal position information in the image can be represented by a position mask map, and the mask value belonging to the abnormal position in this position mask map is the first value (such as 1), and the mask value not belonging to the abnormal position is the second value (such as 0).

[0144] Based on this, the present application can respectively obtain the abnormal position mask map of the image to be classified and the target position mask map of the known abnormality in the corresponding target image according to the abnormal position information of the image to be classified and the target position information of the known abnormality in the target image set. Then, fuse the target position mask maps of each target image in the target image set, that is, sum the position mask values at the same position in each target image to obtain the density value at this position, thereby obtaining the target position density map of the known abnormality in the target image set. Thus, according to the target position density map and the abnormal position mask map, determine the matching degree of the abnormal position between the image to be classified and the target image set.

[0145] It should be understood that for the target position density map of the target image set, it can also be obtained in this way during the construction of the decision tree, so that during the calculation of this matching degree, directly read the stored target position density map of the target image set without online calculation to improve the image classification efficiency.

[0146] Optionally, assume that the target position density map of the known abnormality in the target image set can be calculated according to the following formula:

[0147] ; (5)

[0148] In formula (5), represents the target position density map of the target image set, d i represents the i-th target image in the target image set, that is ; mask(d i ) represents the target position mask map of the known abnormality of the i-th target image, where the pixel value of the image area where the known abnormality is located is 1, and the pixel value of the image area where the known abnormality is not located in this target image is 0. Similarly, the abnormal position mask map of the image to be classified calculated according to this method can be denoted as . For the meanings represented by other letters in the formula, reference can be made to the corresponding explanations of the above formula.

[0149] Based on this, the matching degree of the abnormal position between the image to be classified and the target image set can be calculated according to formula (6), which can be denoted as M pos :

[0150] ; (6)

[0151] In formula (6), ∩ represents the intersection operation, which is the set of common elements in two sets of data. In this embodiment, it represents obtaining the set of mask values of the common abnormal positions between the abnormal position mask map and the target position density map; ∪ represents the union operation, which is the set of all elements in two sets of data. In this embodiment, it represents obtaining the set of mask values of all abnormal positions between the abnormal position mask map and the target position density map. max() represents taking the maximum mask value of each pixel position in the image region where the known abnormality in the target image set is located, that is, the maximum value of each mask value in the target position density map; min() represents taking the minimum mask value of each pixel position in the image region where the known abnormality in the target image set is located, that is, the minimum value of each mask value in the target position density map.

[0152] It can be seen that from formula (6), the larger the proportion of the coverage area of the image region where the known abnormality in the target image set is located in the abnormal region of the image to be classified, the higher the matching degree of the abnormal position. The content on the right side of the dot product operation on the right side in formula (6) can represent the normalized range of the known abnormality in the target image set. Multiplying it by the proportion of the coverage area realizes the normalization of the proportion of the coverage area and obtains the matching degree of the abnormal position. It should be noted that regarding the implementation method of step S42, it includes but is not limited to the calculation method represented by formula (6). It can also determine the matching degree of the abnormal position between the image to be classified and the target image set based on the same position information or position difference between the abnormal position information and the target position distribution. The implementation process is not described in detail in this application.

[0153] Step S43: Determine the matching degree of the abnormal size between the image to be classified and the target image set according to the abnormal size value of the image to be classified, as well as the size expectation value and size standard deviation of each target size value of the known abnormalities in the target image set;

[0154] The implementation method of step S43 is similar to the principle of the implementation method of step S41. For a known abnormality represented by the target image set, the aspect ratio of the length and width of the minimum bounding box circumscribing the known abnormal region (that is, the detection box of the known abnormality in the target image, which can be the smallest rectangle completely containing the known abnormal region) can be calculated. The length and width can be obtained by counting the number of pixels of the corresponding boundary, or by calculating the difference between the minimum and maximum coordinates of the bounding box in the horizontal and vertical directions, etc. This aspect ratio of the length and width of the boundary box reflects the shape characteristics of the known abnormal region in the target image set. At this time, the aspect ratio of the length and width of the abnormal region determined by the abnormal size value of the image to be classified can also be used in the same way, thereby reflecting the abnormal shape characteristics of the image to be classified. Then, by comparing the differences between the shape characteristics of the known abnormalities corresponding to the target image set and the abnormal shape characteristics of the image to be classified, the matching degree of the abnormal size between the image to be classified and the target image set, that is, the matching degree in terms of morphological characteristics, is determined.

[0155] In the above process of analyzing the differences in shape features, the expected value and standard deviation of the aspect ratio of the known abnormal regions (i.e., the above-mentioned bounding boxes) in the target image set can be calculated respectively, that is, the size expected value and size standard deviation of the target size values. According to the calculation relationship between the abnormal pixel values, pixel expected values, and pixel standard deviations expressed by formula (4) and the matching degree on the abnormal color, the obtained abnormal size values, size expected values, and size standard deviations are calculated to obtain the matching degree of the image to be classified and the target image set in terms of abnormal size. Among them, the size expected value and size standard deviation of the target size values of the known abnormalities can be obtained and stored in advance, or can be calculated online, and this application does not limit this.

[0156] Step S44: Determine the matching degree of the image to be classified and the target image set in terms of abnormal edge information based on the abnormal edge information features of the image to be classified and the respective target edge information features of the known abnormalities in the target image set.

[0157] In practical applications, the edge information of the abnormal regions in the image can accurately reflect the shape and texture features of the abnormal regions. This application can adopt one or more edge detection algorithms such as Canny, Sobel, Prewitt, Laplacian, etc. to extract the edge information features of the abnormal regions in the image, such as edge intensity, direction, length, and shape. This application records the edge information features of the abnormal regions extracted from the image to be classified as abnormal edge information features, and records the edge information features of the known abnormal regions extracted from the target image as target edge information features. This application does not elaborate on the operation principles of each edge detection algorithm one by one. Only the Sobel operator is used as an example for illustration below.

[0158] The Sobel operator is a gradient-based edge detection method that detects edges by calculating the horizontal and vertical gradients of the image. Among them, the horizontal gradient G x and the vertical extraction G y are calculated as follows:

[0159] ; (7)

[0160] ; (8)

[0161] In formulas (7) and (8), H can represent the matrix representation of each target image in the target image set. After that, through this calculation method, the first-order gradient matrix of the corresponding target image can be obtained to represent the target edge information features.

[0162] Afterwards, since both the abnormal edge information features and the target edge information features of each target image are matrices, a suitable similarity algorithm, such as cosine similarity, Euclidean distance, Pearson correlation coefficient, or other matrix similarity algorithms, can be used to obtain the similarity of the image to be classified and the corresponding target image in terms of abnormal edge information (which can also be called the matching degree, and the higher the similarity, the higher the corresponding matching degree). Afterwards, from the similarities corresponding to each target image, the minimum similarity can be selected and determined as the matching degree of the image to be classified and the target image set in terms of abnormal edge information.

[0163] Step S45: Determine the matching degree of the image to be classified and the target image set in terms of abnormal high-dimensional features based on the abnormal high-dimensional feature values of the image to be classified and the abnormal high-dimensional feature values of each target image with known abnormalities in the target image set.

[0164] In the embodiments of the present application, since the high-dimensional feature values of the abnormal regions in the image can describe the complex characteristics of the abnormal regions, such as high-level semantic information and descriptions of texture, shape, color distribution, etc., so as to more accurately understand the abnormalities in the image and improve the accuracy of abnormal classification. Therefore, the present application can extract the high-dimensional features of the abnormal regions in the image based on a deep learning model, and this deep learning model can be trained by neural networks such as convolutional neural networks or Vision Transformer (visual transformer). The present application does not limit the network structure of the deep learning model, and the high-dimensional features of the image can also be extracted through known large models. The present application does not limit the extraction method of the high-dimensional features.

[0165] Based on this, after the present application obtains the abnormal high-dimensional feature values of the image to be classified and the abnormal high-dimensional feature values of each target image with known abnormalities in the target image set (which can be obtained and stored during the construction of the decision tree classifier or can also be obtained online during the inference process), a suitable similarity algorithm (which can include but is not limited to the several similarity algorithms listed above, and other feature vector similarity calculation methods, such as other distance calculation methods, etc.) can also be used to calculate the similarity between the abnormal high-dimensional feature values and the known abnormal target high-dimensional feature values of each target image, and then based on the obtained similarities, determine the matching degree of the image to be classified and the target image set in terms of abnormal high-dimensional features, such as selecting the minimum similarity from the similarities and determining it as the matching degree of the image to be classified and the target image set in terms of abnormal high-dimensional features, etc.

[0166] It should be noted that during the construction of the decision tree classifier, the categories and quantities of abnormal attributes of each node determined based on prior knowledge include, but are not limited to, the content described in Embodiment 3. According to actual needs, at least one abnormal attribute can be selected from the 5 abnormal attributes described in Embodiment 3 to implement the construction of the decision tree classifier. Generally, in order to improve the classification accuracy of the decision tree classifier, multiple (such as any two or three or four combinations or 5, etc.) abnormal attributes are usually selected, and the attribute content can be determined based on prior knowledge.

[0167] Of course, other categories of abnormal attributes different from the abnormal attributes listed above can also be combined to implement the construction of the decision tree classifier. In the case where the prior knowledge changes and affects the important attributes of the features of the known abnormalities, the abnormal attributes can also be updated to construct a new decision tree classifier. The image classification process based on the thus constructed decision tree classifier is similar, and the present application does not give detailed examples one by one. Moreover, the present application does not limit the order of obtaining the matching degrees of different abnormal attributes. When resources are sufficient, the processes of obtaining the matching degrees of each abnormal attribute can be executed in parallel to improve the image classification efficiency, or it can be, but not limited to, Figure 4 sequentially obtaining the matching degrees of each abnormal attribute as shown.

[0168] After obtaining the matching degrees of the image to be classified and the target image set on each abnormal attribute according to the method described above, it is possible to comprehensively determine whether the image to be classified belongs to the known abnormality corresponding to the target image set, so as to effectively determine the unknown abnormality. The implementation process can refer to the description of the corresponding part of the above embodiment, and this embodiment will not be elaborated here.

[0169] It can be seen that in the inference process of the decision tree classifier of the present application, by matching the features of each abnormal attribute between the image to be classified and the target image set, the difference between the abnormality of the image to be classified on the corresponding abnormal attribute and the known abnormality of the target image set is determined. Since the higher the matching degree, the smaller the difference on the corresponding abnormal attribute, the higher the probability that this abnormality belongs to this known abnormality. Compared with the absolute classification result directly based on the decision tree, that is, the known abnormality corresponding to the target image set determined by the matching of each abnormal feature, the present application combines the matching degrees on each abnormal attribute to be able to comprehensively and accurately obtain the difference between the abnormality of the image to be classified and the known abnormality of the target image set, and effectively determine the unknown abnormality of the image to be classified, that is, the image to be classified and the target image set do not belong to the same abnormal category, thereby improving the image classification accuracy.

[0170] Refer to Figure 5 , which is a schematic flowchart of an image classification method provided in Embodiment 4 of the present application. This embodiment can refine the description of the construction process of the decision tree classifier described in the above embodiments, such as Figure 5As shown in the figure, the implementation method of constructing a decision tree classifier based on the prior knowledge of each known anomaly in the industrial detection scenario may include but is not limited to the following steps:

[0171] Step S51: Obtain the total set of images in the industrial detection scenario; the total set of images contains multiple images of different known anomalies;

[0172] In the embodiments of the present application, the training data set can be constructed by using each historical image with a determined anomaly category in the industrial detection scenario, denoted as the total set of images in the industrial detection scenario, which can be represented as D, then D = {d1, d2, ……, d n}, d i represents the i-th image in the total set of images, i = 1, 2, ……, n, and n is the total number of images contained in the total set of images. The known anomalies of each image in this total set of images can be configured with corresponding anomaly labels. Assuming there are m known anomalies in total, the formed anomaly label set can be represented as D = {D1, D2, ……, D m}. Among them, for the known anomalies of each image, they can be determined by anomaly recognition algorithms (such as anomaly recognition based on clustering algorithms / machine learning / deep learning, etc., the implementation process of which is not detailed in this application) or manual annotation methods, etc. The anomaly labels can be the known anomaly names or corresponding identifiers or numbers configured one by one, etc., and can be configured in combination with the representation methods of each known anomaly in the industrial detection scenario. This application does not limit the anomaly recognition method and its anomaly label representation method for known anomalies.

[0173] In an alternative implementation method, in order to facilitate determining the characteristics of each anomaly attribute in the anomaly area of the image, this application can determine the position information of the recognized anomaly area during the anomaly recognition process of the image, that is, the position information of the image area where the known anomaly is located in this image, or the position information of the known anomaly manually annotated in the image, etc., and associate and store the configuration contents such as anomaly labels and position information with the corresponding images for reading and use during the subsequent construction process of the decision tree classifier.

[0174] Step S52: Determine the anomaly attributes for identifying known anomalies and the prior weights corresponding to each anomaly attribute according to the prior knowledge of each known anomaly in the industrial detection scenario;

[0175] Regarding the prior knowledge of various known anomalies in the industrial inspection scenario, the prior knowledge can be determined by analyzing the historical data of the industrial inspection scenario to identify each known anomaly category and its characteristics. For example, collect the abnormal images detected in the past, including information such as defect type, location, and shape. It can also be obtained by acquiring the experiential knowledge of industry experts in the industry where the industrial inspection scenario is located (such as through interviews and questionnaires, and by accessing the personal knowledge bases of experts), so that industry experts can provide an intuitive understanding and judgment of the anomaly category and its characteristics based on their long-term work experience. Or obtain relevant feature information of known anomalies from the literature and industry standards in the industry to form prior knowledge, etc.

[0176] In addition, the present application can also generate images of various known anomalies through simulation tests to obtain prior knowledge. For example, in a laboratory environment, artificial faults (such as known anomalies like scratches, cracks, and stains) are created to generate abnormal images, different production environments (such as temperature, humidity, and light) are simulated, the performance of anomalies under different conditions is observed, high-precision detection equipment is used to collect image data in the simulation experiment, and the characteristics of known anomalies are recorded to form prior knowledge. Of course, the present application can also extract relevant feature information of known anomalies from a large amount of data through machine learning and data analysis methods to form prior knowledge. Of course, the present application can also adopt, but is not limited to, two or more of the above-mentioned acquisition methods to combine and obtain prior knowledge in the industrial inspection scenario, etc. The present application does not limit the content and acquisition method of the prior knowledge of each known anomaly in the industrial inspection scenario.

[0177] In the embodiments of the present application, it can be understood in combination with the content of the prior knowledge described above that this prior knowledge characterizes the characteristics of the attributes (different from non-abnormal attributes) of various known anomalies existing in the industrial inspection scenario, as well as the importance of this attribute feature in identifying the known anomaly. Therefore, by analyzing the prior knowledge, at least one abnormal attribute can be determined from the attributes of the known anomaly. To improve the classification accuracy, multiple attributes are usually selected as abnormal attributes, such as the abnormal color, abnormal location, abnormal size, abnormal edge information, and abnormal high-dimensional features exemplified above.

[0178] At the same time, the prior weight of the abnormal attribute can be determined based on the importance (i.e., the degree of importance) of each abnormal attribute in identifying the corresponding known anomaly, so as to indicate the importance of the corresponding abnormal attribute in identifying the known anomaly to which the image belongs through this prior weight. The higher the degree of importance, the greater the prior weight of the corresponding abnormal attribute. The present application does not limit the method for determining the prior weight of each abnormal attribute, which can be configured by industry experts or calculated based on a linear function. The implementation process can refer to the description of the corresponding part of the following embodiments.

[0179] After that, this application can recursively split the total set of images that make up the initial decision tree according to the known anomalies, anomaly attributes, and prior weights of each of the multiple images (i.e., all images in the total set of images), until the split image sets belong to the same known anomaly, obtaining a decision tree classifier. Among them, the total set of images can be the nodes of the initial decision tree. Based on the information gain rate and prior weight of each anomaly attribute, the selection of the splitting attribute in this total set of images is realized. Accordingly, the total set of images is divided into at least two subsets, which are created as the intermediate nodes of the decision tree. Then, the selection and division of the splitting attribute are repeated for each divided subset to complete the construction of the decision tree classifier. The construction process of this decision tree classifier can refer to the description in steps S53 - S55 as follows.

[0180] Step S53, determine the position information corresponding to the known anomalies of each of the multiple images, and determine the target features corresponding to each anomaly attribute of the corresponding image;

[0181] Combined with the above - mentioned relevant descriptions of anomaly attributes and their features, such as the pixel values of abnormal colors, the position mask map of abnormal positions, the aspect ratio of the area of abnormal sizes, the abnormal edge information features, etc., using the corresponding feature extraction algorithms, the features of the abnormal regions of each image in the total set of images are extracted. The features of each anomaly attribute extracted are recorded as the target features of the corresponding anomaly attribute. This application does not elaborate on the feature extraction method.

[0182] Step S54, determine the information gain rate of each anomaly attribute corresponding according to the anomaly labels of each of the multiple images and the target features of each anomaly attribute;

[0183] In the embodiments of this application, the information gain rate corresponding to the anomaly attribute in each image can be calculated, and the best splitting attribute can be selected from all the anomaly attributes in this image set to improve the classification performance of the decision tree. This application does not limit the calculation method of the information gain rate. The following formula is only an example and does not constitute a limitation on the information gain rate calculation method.

[0184] ; (9)

[0185] ; (10)

[0186] ; (11)

[0187] ; (12)

[0188] ; (13)

[0189] In the above formula, Info() represents the information entropy operation, represents the anomaly attribute A of each image in the total set of images Di The classification information amount, i.e., the calculation result of the information entropy of the i-th abnormal attribute (A i ), where c represents the number of features corresponding to this abnormal attribute A i , which can be the number of discrete feature values corresponding to the abnormal attribute A i , and m represents the total number of known abnormal categories in the total image set. represents the information amount of this node (total image set D) calculated through information entropy operation when the total image set D is divided according to the abnormal attribute Ai. represents the information amount of the total image set D, which can be obtained by calculating the features of each abnormal attribute in the total image set D through the information entropy operation method. This application does not elaborate on each information entropy calculation method.

[0190] Among them, for the discrete features corresponding to each abnormal attribute (i.e., the values of the corresponding abnormal attributes, which can also be called discrete feature values), they can be obtained by discretizing the value range of the corresponding abnormal attribute (which is usually a continuous feature value). Optionally, after determining the value range where the target features corresponding to each abnormal attribute in the total image set are located, or determining the feature value range corresponding to each abnormal attribute based on prior knowledge, the value ranges corresponding to different abnormal attributes can be discretized respectively through the K (which is a hyperparameter and can be flexibly configured according to the actual situation or take the default value, etc., and this application does not limit the value of K) -means clustering method to obtain different target features corresponding to the corresponding abnormal attributes, that is, discrete feature values. For example, each cluster center (mean value) after clustering is used as a target feature, and after several iterations of clustering until the cluster center no longer changes or reaches the maximum number of iterations, etc., this application does not elaborate on the specific method of realizing continuous numerical discretization based on the K -means clustering method and is not limited to this data discretization method.

[0191] After obtaining the corresponding information amounts through the information entropy operation methods represented by formula (11), formula (12), and formula (13) respectively, the information gain corresponding to the abnormal attribute A i when it is selected as the splitting attribute can be determined through the difference operation method of formula (10). i It can be denoted as , which can be used to measure the useful degree index of this abnormal attribute A i in the abnormal classification task in the industrial detection scenario, indicating how much the uncertainty of the abnormal category is reduced after knowing the features of this abnormal attribute A i . After that, the information gain rate corresponding to the abnormal attribute A i can be determined through the ratio operation method represented by formula (9), denoted as , that is, the information gain corresponding to the abnormal attribute A i divided by the information amount of the abnormal attribute A iThe ratio between the information entropy.

[0192] Step S55: Recursively split the total image set according to the information gain ratio and the prior weight until the split image sets belong to the same known anomaly, obtaining a decision tree classifier.

[0193] Following the above analysis, since the images in the total image set D do not belong to the same anomaly category, after calculating the information gain ratio corresponding to each anomaly attribute according to the above method, compared with only based on the information gain ratio, when selecting the best split attribute of the total image set D, that is, the anomaly attribute corresponding to the maximum information gain ratio as the split attribute, this application also takes into account the prior weights corresponding to each anomaly attribute represented by prior knowledge, that is, the importance of each anomaly attribute for identifying the known anomaly to which the image belongs, to realize the selection of the split attribute, improving the accuracy and reliability of the split attribute selection, and thus improving the classification performance of the constructed classifier decision tree.

[0194] Optionally, this application can directly sum the information gain ratio and the prior weight of the same anomaly attribute, and determine the obtained value as the selection index of this anomaly attribute, which is equivalent to using this prior weight to update the information gain ratio of the corresponding anomaly attribute, so as to use the updated information gain ratio to select the split attribute. The implementation process of this can adopt the following formula:

[0195] ; (14)

[0196] In formula (14), argmax() can represent selecting the anomaly attribute corresponding to the maximum value as the split attribute from the sum of the information gain ratio and the prior weight corresponding to each anomaly attribute, denoted as A * . However, it is not limited to the implementation method of split attribute selection described in formula (14). Thus, it can be seen that in the process of this application selecting the split attribute for each image set, the higher the importance of the anomaly attribute for identifying the known anomaly to which the image belongs, the higher the probability that this anomaly attribute is selected as the split attribute, and the higher the information gain ratio of the anomaly attribute, the higher the probability that this anomaly attribute is selected as the split attribute. For multiple anomaly attributes with similar information gain ratios, since this application also combines the prior weights of each anomaly attribute, the combination of the two can more accurately distinguish which of these multiple anomaly attributes is more suitable to be selected as the split attribute, improving the accuracy and reliability of the split attribute selection.

[0197] Divide the total image set D according to the selected splitting attribute, such as calculating the information gain or Gini impurity of each feature of the splitting attribute, and selecting the feature corresponding to the maximum information gain or the minimum Gini impurity as the splitting point to achieve the division of D. After that, each divided subset (image set) is used as an intermediate node of the decision tree. The feature ranges corresponding to the splitting attributes included in different intermediate nodes are different, and they are used as the feature classification conditions for the intermediate nodes. After that, for each divided subset, the information gain rate of each abnormal attribute therein can be recalculated according to the method described above (in this calculation process, the corresponding subset is equivalent to the total image set D in the above information gain rate calculation formula), and the splitting attribute of the subset is selected by combining the prior weights, and the subset is divided according to the splitting attribute, and so on recursively, gradually creating the intermediate nodes of the decision tree until each of the split subsets contains images of the same abnormal category, forming the leaf nodes of the decision tree, and completing the construction of the decision tree classifier.

[0198] It can be seen that the above total image set and the image sets after each split are used to determine the nodes of the decision tree classifier. The hierarchical relationship between the nodes is determined according to the recursive splitting order of the total image set. During the process of splitting the image set of the corresponding node according to the splitting attribute, the feature classification conditions of the corresponding sub-nodes obtained by splitting can be determined according to the selected splitting point, so as to realize the feature matching of the abnormal features of the image to be classified with the features of each layer of nodes during the inference process of the decision tree classifier, and determine the leaf node of the path to which the abnormal feature belongs, that is, realize the preliminary abnormal classification of the image to be classified. The implementation process can refer to the inference process described in the above embodiments, and this embodiment will not be elaborated here.

[0199] In some embodiments, for the process of obtaining the prior weights corresponding to the above different abnormal attributes, the computer device can determine the prior weights of the corresponding abnormal attributes in response to the weight allocation operation of each abnormal attribute of the known abnormalities in the industrial detection scenario, that is, the industry experts can, according to their personal work experience, configure the prior weights of each abnormal attribute in the configuration interface output by the computer device. The implementation process will not be elaborated in this application.

[0200] In other embodiments, in order to reduce the subjective influence of the expert in configuring the prior weights, the total image set can be analyzed to calculate the prior weights of each abnormal attribute. In this implementation process, the standard deviation of the target features of each known abnormality of the total image set on the same abnormal attribute can be obtained, and the prior weights of the corresponding abnormal attributes can be determined according to the standard deviations corresponding to the abnormal attributes. Among them, the abnormal attribute with a larger standard deviation corresponds to a smaller prior weight.

[0201] It can be seen that the present application can sort the standard deviations corresponding to each abnormal attribute, for example, sort them in ascending order. After normalization (it can also be normalized first and then sorted, and the present application does not limit the execution order of the two), the prior weights corresponding to the normalized standard deviations of different abnormal attributes can be calculated according to a linear function (such as a decreasing linear function for reverse order, etc.). It should be noted that the present application does not limit the type and representation method of this linear function, which can be determined according to experience and experiments.

[0202] In addition, it should be understood that after the prior knowledge in the industrial detection scenario changes significantly, that is, the importance of different abnormal attributes represented by the prior knowledge for identifying the abnormal category of the image changes, the prior weights of each abnormal attribute can be updated in a timely manner according to the method described above, and then the decision tree classifier can be reconstructed according to the method described above to ensure the image classification accuracy based on the decision tree classifier. Optionally, the present application can also verify whether the prior weights of each abnormal attribute have changed periodically or according to certain rules. If they have changed, the decision tree classifier is updated.

[0203] To sum up, in the image classification method proposed in the present application, referring to Figure 6 the flowchart of an image classification method provided in Embodiment 5 of the present application shown, in the process of constructing a decision tree classifier for the industrial detection scenario, after determining the abnormal attributes and their corresponding prior weights of each known abnormality according to the prior knowledge, in the process of recursively splitting the total image set in the industrial detection scenario, after calculating the information gain rate of each abnormal attribute in the split image set, the split attribute will be accurately selected by combining the prior weights of each abnormal attribute, improving the splitting accuracy of the image set, thereby improving the classification performance of the constructed decision tree classifier, solving the problem of low classification accuracy caused by the manually constructed rule-based classifier, and compared with the implementation method of only selecting the split attribute based on the information gain rate to split the image set to construct a decision tree, the present application introduces the prior knowledge in the industrial detection scenario, further improving the classification accuracy.

[0204] Moreover, in the inference process based on the decision tree classifier, as Figure 6 shown, after the present application determines the target image set corresponding to the abnormal features of each abnormal attribute in the image to be classified, that is, a leaf node of the decision tree, it will further calculate the matching degree between each abnormal feature and the target feature of the same abnormal attribute in the target image set, and verify whether the image to be classified and the target image set belong to the same abnormal category by combining the subsequent weights of each abnormal attribute represented by the decision tree classifier, not only improving the classification accuracy of known abnormalities, but also being able to effectively determine the unknown abnormalities of the image to be classified, improving the accuracy and reliability of image abnormal classification.

[0205] In some embodiments, when the target abnormal category of the image to be classified is an unknown abnormality, corresponding abnormality prompt information can be output to notify professional experts to promptly identify and label the abnormality label of the image to be classified, or send the image to be classified to a large model or a high-performance image classification model to determine the abnormality label of the unknown abnormality. After determining several unknown abnormalities, their images and abnormality labels can be used to update the above-mentioned total image set, and a decision tree classifier can be reconstructed according to the method described above to improve the accuracy of abnormality classification based on the decision tree classifier in industrial detection scenarios.

[0206] The above introduces an image classification method provided by an embodiment of the present application. The following will introduce an apparatus for executing the above image classification method.

[0207] Refer to Figure 7 , which is a schematic structural diagram of an image classification apparatus provided by an embodiment of the present application. This image classification apparatus can be applicable to computer devices, such as Figure 7 shown, this image classification apparatus may include:

[0208] An image acquisition module 71, configured to acquire an image to be classified in an industrial detection scenario;

[0209] A feature extraction module 72, configured to extract features from the image to be classified to obtain abnormal features corresponding to each abnormal attribute in the image to be classified;

[0210] An abnormal classification module 73, configured to input the abnormal features into a pre-constructed decision tree classifier. After determining a target image set containing each of the abnormal features in the decision tree classifier, the target abnormal category of the image to be classified is determined by calculating the matching degree between the abnormal features and the target features of the same abnormal attribute in the target image set;

[0211] Wherein, the decision tree classifier is constructed based on the prior knowledge of each known abnormality in the industrial detection scenario, and the abnormal attribute is an attribute that the known abnormality has determined according to the prior knowledge; the target abnormal category is an unknown abnormality or a known abnormality to which the target image set belongs.

[0212] In a possible implementation, the abnormal classification module 73 may include:

[0213] A target feature determination unit, configured to determine the target features of different abnormal attributes of each target image in the target image set;

[0214] A matching degree determination unit, configured to calculate the matching degree between the abnormal features and the target features corresponding to the same abnormal attribute, and determine the matching degree between the image to be classified and the target image set in this abnormal attribute;

[0215] A target abnormal category determination unit, configured to determine a target abnormal category of the image to be classified according to the matching degree;

[0216] Wherein, the higher the matching degree, the higher the probability that the image to be classified and the target image set belong to the same abnormal category; if the image to be classified and the target image set do not belong to the same abnormal category, the target abnormal category is the unknown abnormality.

[0217] Optionally, the target abnormal category determination unit may include:

[0218] A posterior weight determination unit, configured to determine a posterior weight of the corresponding abnormal attribute according to the classification importance degree of each abnormal attribute represented by the decision tree classifier;

[0219] A probability prediction unit, configured to determine a probability that the image to be classified and the target image set belong to the same abnormal category according to the matching degree corresponding to each abnormal attribute and the posterior weight;

[0220] A first determination unit, configured to determine a target abnormal category of the image to be classified according to the probability.

[0221] In an alternative implementation, the matching degree determination unit of the above abnormal classification module 73 may include at least one of the following matching calculation units:

[0222] A first matching calculation unit, configured to determine a matching degree in abnormal color between the image to be classified and the target image set according to the abnormal pixel values included in the abnormal feature, and the pixel expected value and pixel standard deviation of each target pixel value of the known abnormality in the target image set;

[0223] A second matching calculation unit, configured to determine a matching degree in abnormal position between the image to be classified and the target image set according to the abnormal position information included in the abnormal feature, and the target position distribution of the known abnormality in the target image set;

[0224] A third matching calculation unit, configured to determine a matching degree in abnormal size between the image to be classified and the target image set according to the abnormal size value included in the abnormal feature, and the size expected value and size standard deviation of each target size value of the known abnormality in the target image set;

[0225] A fourth matching calculation unit, configured to determine a matching degree in abnormal edge information between the image to be classified and the target image set according to the abnormal edge information feature included in the abnormal feature, and each target edge information feature of the known abnormality in the target image set;

[0226] A fifth matching calculation unit, configured to determine a matching degree between the image to be classified and the target image set in terms of abnormal high-dimensional features according to the abnormal high-dimensional feature values included in the abnormal features and the high-dimensional feature values of each known abnormal target in the target image set.

[0227] In summary of the descriptions of the above embodiments, the construction module for constructing the decision tree classifier according to the prior knowledge of each known abnormality in the industrial detection scenario may include:

[0228] An overall image set acquisition unit, configured to acquire an overall image set in the industrial detection scenario; the overall image set includes multiple images with different known abnormalities;

[0229] A second determination unit, configured to determine abnormal attributes for identifying the known abnormalities and prior weights corresponding to the abnormal attributes according to the prior knowledge of each known abnormality in the industrial detection scenario; the prior weights indicate the importance degrees of the corresponding abnormal attributes for identifying the known abnormality to which the image belongs.

[0230] A construction unit, configured to recursively split the overall image set that constitutes the initial decision tree according to the known abnormalities, abnormal attributes, and prior weights of the multiple images until the split image sets belong to the same known abnormality, so as to obtain a decision tree classifier;

[0231] Wherein, the overall image set and the image sets after each split are determined as nodes of the decision tree classifier, and the hierarchical relationship between the nodes is determined according to the recursive split order of the overall image set.

[0232] Optionally, each image in the overall image set is configured with an abnormal label corresponding to the known abnormality of the image and position information of the image area where the known abnormality is located; based on this, the above construction unit may include:

[0233] A third determination unit, configured to determine target features corresponding to the abnormal attributes of the corresponding image according to the position information corresponding to the multiple images;

[0234] An information gain rate determination unit, configured to determine the information gain rates of the abnormal attributes according to the abnormal labels corresponding to the multiple images and the target features of the abnormal attributes;

[0235] A recursive split unit, configured to recursively split the overall image set according to the information gain rate and the prior weight until the split image sets belong to the same known abnormality.

[0236] Optionally, the above second determination unit may include:

[0237] A fourth determination unit, configured to determine a prior weight corresponding to the abnormal attribute in response to an operation of assigning weights to each abnormal attribute of known abnormalities in an industrial detection scenario;

[0238] A fifth determination unit, configured to determine a prior weight corresponding to the abnormal attribute according to the standard deviation of the target features of each known abnormality in the image set on the same abnormal attribute; the larger the standard deviation of the abnormal attribute, the smaller the corresponding prior weight.

[0239] In an optional implementation, the above second matching calculation unit may include:

[0240] A position mask map obtaining unit, configured to obtain an abnormal position mask map of the image to be classified and a target position mask map of a known abnormality in the corresponding target image respectively according to the abnormal position information included in the abnormal feature and the target position information of the known abnormality in the target image set;

[0241] A target position density map obtaining unit, configured to fuse the respective target position mask maps corresponding to the target image set to obtain a target position density map of the known abnormality in the target image set;

[0242] A sixth determination unit, configured to determine the matching degree between the image to be classified and the target image set in terms of abnormal positions according to the target position density map and the abnormal position mask map;

[0243] Wherein, in each position mask map, the mask value belonging to the abnormal position is a first numerical value, and the mask value not belonging to the abnormal position is a second numerical value.

[0244] Optionally, the above first matching calculation unit may include:

[0245] An obtaining unit, configured to obtain a pixel expected value and a pixel standard deviation of each target pixel value of a known abnormality in the target image set;

[0246] A pixel difference determination unit, configured to determine a pixel difference between the image to be classified and the target image set in terms of abnormal color according to the pixel expected value and the abnormal pixel value included in the abnormal feature;

[0247] A seventh determination unit, configured to determine the matching degree between the image to be classified and the target image set in terms of the abnormal color according to the pixel standard deviation and the pixel difference.

[0248] An embodiment of the present application also provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a computer device, the computer device can implement any image classification method provided by the embodiment of the present application.

[0249] An embodiment of the present application also provides a computer program product, including computer-readable instructions, which, when running on a computer device, enable the computer device to implement any one of the image classification methods provided by the embodiments of the present application.

[0250] An embodiment of the present application also provides a computer device. Refer to Figure 8 FIG. which is an optional hardware structure diagram of a computer device applicable to the image classification method proposed in the embodiment of the present application. The computer device may be the terminal 110 or the server 120 listed above. When the computer device is the server 120, as Figure 8 shown, the computer device may include: at least one communication component 81, at least one memory 82, and at least one processor 83, where:

[0251] The at least one communication component 81, the at least one memory 82, and the at least one processor 83 can communicate through a bus. The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, Figure 8 only a single bidirectional line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0252] The communication component 81 can be used to obtain the image to be classified in the industrial inspection scenario. It can also be used to implement data or instruction transmission between the internal components of the computer device. In the embodiment of the present application, the communication component 81 may include communication components corresponding to wireless communication methods such as wifi, bluetooth, 5G / 6G, etc., so that the computer device can implement data transmission with other devices through the communication component. It may also include one or more interfaces supporting wired communication methods, such as general-purpose input / output (GPIO) interfaces, USB interfaces, universal asynchronous receiver / transmitter (UART) interfaces, etc., to implement data transmission between the internal components of the computer device. The present application does not limit the composition structure of the communication component 81 to implement this function and its corresponding communication transmission mechanism, which can be determined according to the situation.

[0253] The memory 82 can be used to store multiple computer instructions for implementing the image classification method proposed in the embodiments of this application; the processor 83 can load and execute the multiple computer instructions stored in the memory 82 to implement each step of the image classification method proposed in the embodiments of this application. The implementation process can refer to the description of the corresponding part in the above method embodiments.

[0254] In the embodiments of this application, the memory 82 can include storage media such as floppy disks, USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs. The processor 83 can include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0255] It should be understood that Figure 8 the structure of the computer device shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the computer device may include more or fewer components than Figure 8 those shown, or combine certain components. Especially when the computer device is the terminal 110, as Figure 9 shown, the computer device may further include one or more input components 84 such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc., and one or more output components 85 such as a liquid crystal display (LCD), a speaker, a vibrator, etc. The input components 84 and the output components 85 can be connected to the bus through the I / O interface 86. According to the functional requirements of the computer device, other components may also be included, which are not elaborated one by one in this application.

[0256] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0257] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by dedicated hardware, etc. That is to say, in the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product in the form of a software product can be stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, etc., including several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the image classification method described in each embodiment of this application.

Claims

1. An image classification method, the method comprising: Obtaining an image to be classified in an industrial inspection scenario; Performing feature extraction on the image to be classified to obtain abnormal features corresponding to each abnormal attribute in the image to be classified; Inputting the abnormal features into a pre-constructed decision tree classifier. After determining a target image set containing each of the abnormal features in the decision tree classifier, by calculating a matching degree between the abnormal features and target features of the same abnormal attribute in the target image set, determining a target abnormal category of the image to be classified; Wherein, the decision tree classifier is constructed based on prior knowledge of each known abnormality in the industrial inspection scenario, and the abnormal attribute is an attribute that the known abnormality has determined according to the prior knowledge; The target abnormal category is an unknown abnormality or a known abnormality to which the target image set belongs.

2. The method according to claim 1, wherein the determining the target abnormal category of the image to be classified by calculating a matching degree between the abnormal features and target features of the same abnormal attribute in the target image set comprises: Determining target features of different abnormal attributes possessed by each target image in the target image set; Calculating a matching degree between the abnormal features and the target features corresponding to the same abnormal attribute to determine a matching degree between the image to be classified and the target image set in this abnormal attribute; Determining the target abnormal category of the image to be classified according to the matching degree; Wherein, the higher the matching degree, the higher the probability that the image to be classified and the target image set belong to the same abnormal category; If the image to be classified and the target image set do not belong to the same abnormal category, the target abnormal category is the unknown abnormality.

3. The method according to claim 2, wherein the determining the target abnormal category of the image to be classified according to the matching degree comprises: Determining a posterior weight of the corresponding abnormal attribute according to the classification importance degree of each abnormal attribute represented by the decision tree classifier; Determining a probability that the image to be classified and the target image set belong to the same abnormal category according to the matching degree corresponding to each abnormal attribute and the posterior weight; Determining the target abnormal category of the image to be classified according to the probability.

4. The method according to any one of claims 1-3, wherein the calculating the matching degree between the abnormal features and target features of the same abnormal attribute in the target image set comprises at least one of the following: Determining a matching degree between the image to be classified and the target image set in abnormal color according to the abnormal pixel values included in the abnormal features, and the pixel expected value and pixel standard deviation of each target pixel value of the known abnormality in the target image set; Determining a matching degree between the image to be classified and the target image set in abnormal position according to the abnormal position information included in the abnormal features, and the target position distribution of the known abnormality in the target image set; Determine the matching degree of the image to be classified and the target image set in terms of abnormal dimensions based on the abnormal dimension values included in the abnormal features, as well as the dimension expected values and dimension standard deviations of the target dimension values of the known abnormalities in the target image set; Determine the matching degree of the image to be classified and the target image set in terms of abnormal edge information based on the abnormal edge information features included in the abnormal features, as well as the target edge information features of the known abnormalities in the target image set; Determine the matching degree of the image to be classified and the target image set in terms of abnormal high-dimensional features based on the abnormal high-dimensional feature values included in the abnormal features, as well as the target high-dimensional feature values of the known abnormalities in the target image set.

5. The method according to any one of claims 1-3, wherein the decision tree classifier is constructed based on the prior knowledge of each known abnormality in the industrial inspection scenario, including: Obtain the total set of images in the industrial inspection scenario; the total set of images includes multiple images with different known abnormalities; Determine the abnormal attributes for identifying the known abnormalities and the corresponding prior weights of each of the abnormal attributes according to the prior knowledge of each known abnormality in the industrial inspection scenario; the prior weights indicate the importance of the corresponding abnormal attributes for identifying the known abnormality to which the image belongs; Recursively split the total set of images constituting the initial decision tree according to the known abnormalities, the abnormal attributes, and the prior weights of the multiple images until the split image set belongs to the same known abnormality, and obtain the decision tree classifier; Wherein, the total set of images and the image set after each split are determined as the nodes of the decision tree classifier, and the hierarchical relationship between the nodes is determined according to the recursive splitting order of the total set of images.

6. The method according to claim 5, wherein each image in the total set of images is configured with an abnormal label corresponding to the known abnormality of the image and the position information of the image area where the known abnormality is located; The recursively splitting the total set of images constituting the initial decision tree according to the known abnormalities, the abnormal attributes, and the prior weights of the multiple images until the split image set belongs to the same known abnormality includes: Determine the target features corresponding to each of the abnormal attributes of the corresponding image according to the position information of the multiple images; Determine the information gain rate of each of the abnormal attributes according to the abnormal labels corresponding to the multiple images and the target features of each of the abnormal attributes; Recursively split the total set of images according to the information gain rate and the prior weights until the split image set belongs to the same known abnormality.

7. The method according to claim 6, wherein determining the prior weights corresponding to each abnormal attribute according to the prior knowledge of each known abnormality in the industrial inspection scenario includes any one of the following: Respond to the weight assignment operation of each abnormal attribute of the known abnormalities in the industrial inspection scenario, and determine the prior weight of the corresponding abnormal attribute; Determine the prior weight corresponding to the abnormal attribute according to the standard deviation of the target features of each known abnormality on the same abnormal attribute in the total set of images; the larger the standard deviation of the abnormal attribute, the smaller the corresponding prior weight.

8. The method according to claim 4, wherein the determining the matching degree of the image to be classified and the target image set in terms of the abnormal position according to the abnormal position information included in the abnormal feature and the target position distribution of the known abnormalities in the target image set comprises: Obtain the abnormal position mask image of the image to be classified and the target position mask image of the known abnormalities in the corresponding target image respectively according to the abnormal position information included in the abnormal feature and the target position information of the known abnormalities in the target image set; Fuse the respective target position mask images corresponding to the target image set to obtain the target position density map of the known abnormalities in the target image set; Determine the matching degree of the image to be classified and the target image set in terms of the abnormal position according to the target position density map and the abnormal position mask image; Wherein, in each position mask image, the mask value belonging to the abnormal position is the first value, and the mask value not belonging to the abnormal position is the second value.

9. The method according to claim 4, wherein the determining the matching degree of the image to be classified and the target image set in terms of the abnormal color according to the abnormal pixel value included in the abnormal feature and the expected value and standard deviation of each target pixel value of the known abnormalities in the target image set comprises: Obtain the pixel expected value and pixel standard deviation of each target pixel value of the known abnormalities in the target image set; Determine the pixel difference between the image to be classified and the target image set in terms of the abnormal color according to the pixel expected value and the abnormal pixel value included in the abnormal feature; Determine the matching degree of the image to be classified and the target image set in terms of the abnormal color according to the pixel standard deviation and the pixel difference.

10. A computer device, the computer device comprising: At least one communication component, at least one memory, and at least one processor, wherein: The communication component is configured to obtain the image to be classified in the industrial detection scenario; The memory is configured to store a plurality of computer instructions; The processor is configured to load and execute the computer instructions to implement the following steps: Extract features from the image to be classified to obtain the abnormal features corresponding to each abnormal attribute in the image to be classified; Input the abnormal features into the constructed decision tree classifier. After determining the target image set including each abnormal feature in the decision tree classifier, determine the target abnormal category of the image to be classified by calculating the matching degree between the abnormal features and the target features of the same abnormal attribute in the target image set; Wherein, the decision tree classifier is constructed according to the prior knowledge of each known abnormality in the industrial detection scenario, and the abnormal attribute is the attribute that the known abnormality has determined according to the prior knowledge; The target abnormal category is an unknown abnormality or a known abnormality to which the target image set belongs.

Images