Target classification method, terminal, storage medium and computer program product

By dividing the attribute information of the image to be detected into two parts and conducting separate detection, the problems of complex and low accuracy of defect classification in the production line are solved, and higher classification accuracy and reduced classification difficulty are achieved.

CN118941858APending Publication Date: 2024-11-12BOE TECHNOLOGY GROUP CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410986540.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the production line, the existing automatic defect classification methods have problems such as complex defect classification and low classification accuracy, mainly due to the large number of defect types and large interference of background information.

Method used

Using the target classification method, the attribute information of the image to be detected is divided into two parts, and the first classification branch and the second classification branch are respectively detected separately. The first classification branch is used to detect the process section where the defect is located. The binary classification branch is used to detect defect types, classify through local images, reduce background information interference and reduce classification difficulty.

Benefits of technology

The accuracy of target classification is improved, the target types in each classification branch is reduced, the number of samples is increased, the difficulty of classification is reduced, and the interference of background information on the classification process is avoided, and the accuracy of defect classification is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941858A_ABST
    Figure CN118941858A_ABST
Patent Text Reader

Abstract

The invention provides a target classification method, a terminal, a storage medium and a computer program product, and the method comprises the steps: obtaining a first region image in a to-be-detected image, the to-be-detected image comprises first attribute information and second attribute information representing a target in a to-be-detected piece, and the first region is a region where the target is located; the first area image or the to-be-detected image is input to a first classification branch, the first area image is input to a second classification branch, the first classification branch performs first target classification and outputs a first target classification result, and the second classification branch performs second target classification and outputs a second target classification result; the first target classification result represents first attribute information, and the second target classification result represents second attribute information; and generating a third target classification result of the to-be-detected image according to the first target classification result and the second target classification result. Compared with the prior art, the target classification complexity is reduced, and the target classification accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and more specifically, to a target classification method, a terminal, a storage medium, and a computer program product. Background Art

[0002] A product production line refers to a system that organically connects a series of processes and equipment, and can achieve continuous processing and production of products according to a certain process flow on the production line. In order to ensure product quality, it is usually necessary to classify defects in the production line so that the process flow or production parameters can be adjusted in time to improve the product yield.

[0003] At present, with the increasing application of machine learning and computer vision, machine learning and computer vision technologies can be used in production lines for automatic defect classification to improve detection speed and quality. However, a production line usually involves a lot of processes, and each process may produce a number of corresponding defects. Therefore, there are many types of defects in a production line. When using existing automatic defect classification methods to classify defects, there are technical problems such as complex defect classification and low defect classification accuracy. Summary of the invention

[0004] The purpose of the present disclosure is to provide a target classification method, a terminal, a storage medium and a computer program product to solve the technical problems of complex target classification and low target classification accuracy in related technologies.

[0005] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0006] The first aspect of the present disclosure provides a target classification method, comprising the following steps:

[0007] Acquire a first area image in an image to be detected, wherein the image to be detected includes first attribute information and second attribute information characterizing a target in the object to be detected, and the first area is an area where the target is located;

[0008] Inputting the first region image or the image to be detected into a first classification branch, and inputting the first region image into a second classification branch, wherein the first classification branch is configured to perform a first target classification on the input first region image or the image to be detected and output a first target classification result, and the second classification branch is configured to perform a second target classification on the input first region image and output a second target classification result, wherein the first target classification result represents the first attribute information, and the second target classification result represents the second attribute information;

[0009] A third target classification result of the image to be detected is generated according to the first target classification result and the second target classification result.

[0010] Optionally, before the step of acquiring the first area image in the image to be detected, the step further includes:

[0011] Acquire a first training set, wherein the first training set includes a plurality of first sample images, wherein the first sample image is an image at a second region in an initial sample image, wherein the initial sample image includes first attribute information and second attribute information characterizing an object in a sample, and the second region is a region where the object is located;

[0012] The initial first target classification model is trained using the first training set to obtain a trained first target classification model, wherein the first target classification model includes the first classification branch and the second classification branch, and the first classification branch is configured to perform a first target classification on the input first area image and output a first target classification result.

[0013] Optionally, the first target classification model includes an image processing module, a first classification module and a second classification module, the image processing module and the first classification module form the first classification branch, the image processing module and the second classification module form the second classification branch, the image processing module includes an input end, a first output end and a second output end, the input end is configured to input the first area image, the first output end is connected to the input end of the first classification branch, the second output end is connected to the input end of the second classification branch, the image processing module is configured to perform convolution and down-sampling processing on the first area image to obtain a second area image, and output the second area image to the first classification module through the first output end, and output the second area image to the second classification module through the second output end, the first classification module is configured to perform a first target classification on the input second area image and output a first target classification result, and the second classification module is configured to perform a second target classification on the input second area image and output a second target classification result.

[0014] Optionally, the initial first target classification model is a convolutional neural network model; and the step of training the initial first target classification model using the training set to obtain a trained first target classification model includes:

[0015] Inputting the first sample image into the initial first target classification model to obtain a first target prediction value and a second target prediction value, wherein the first target prediction value represents a prediction value of a first attribute in the first sample image, and the second target prediction value represents a prediction value of a second attribute in the first sample image, and the network parameters in the initial first target classification model have initial values;

[0016] Acquire a first target true value and a second target true value of the first sample image, wherein the first target true value represents a true value of a first attribute in the first sample image, and the second target true value represents a true value of a second attribute in the first sample image;

[0017] Calculating a first loss value according to the first target true value and the first target predicted value, calculating a second loss value according to the second target true value and the second target predicted value, and performing weighted summation of the first loss value and the second loss value according to a preset weight coefficient to obtain a third loss value;

[0018] The network parameters in the initial first target classification model are updated according to the third loss value and the back propagation algorithm until the third loss value converges.

[0019] Optionally, the preset weight coefficient includes a first weight coefficient corresponding to the first loss value and a second weight coefficient corresponding to the second loss value, and the values ​​of the first weight coefficient and the second weight coefficient are positively correlated with the classification difficulty of the first attribute and the second attribute.

[0020] Optionally, the first weight coefficient is the number of categories of the first attribute, and the second weight coefficient is the number of categories of the second attribute; or

[0021] The first weight coefficient is the accuracy of the second target classification result in the first target classification model, and the second weight coefficient is the accuracy of the first target classification result in the first target classification model.

[0022] Optionally, the first target classification model is a ResNeXt convolutional neural network model, the image processing module includes an input layer, a convolution layer, a first pooling layer, a first group convolution block and a second group convolution block connected in sequence, the first classification module and the second classification module have the same network structure, respectively including a third group convolution block, a fourth group convolution block, a second pooling layer and a fully connected layer connected in sequence, the output end of the second group convolution block is connected to the input end of the third group convolution block, the fully connected layer of the first classification module outputs the first target classification result, and the fully connected layer of the second classification module outputs the second target classification result.

[0023] Optionally, before the step of acquiring the first area image in the image to be detected, the step further includes:

[0024] Acquire a second training set and a third training set, wherein the second training set includes a plurality of second sample images, and the third training set includes a plurality of third sample images, wherein the second sample images include first attribute information and second attribute information characterizing an object in a sample piece, and the third sample images are images of a third region in the second sample images, and the third region is a region where the object is located;

[0025] The initial second target classification model is trained by the second training set to obtain a trained second target classification model, the second target classification model is used as the first classification branch, and the first classification branch is configured to perform a first target classification on the input image to be detected and output a first target classification result;

[0026] The initial third target classification model is trained using the third training set to obtain a trained third target classification model, and the third target classification model serves as the second classification branch.

[0027] Optionally, the second target classification model and the third target classification model are ResNeXt convolutional neural network models or DenseNet convolutional neural network models.

[0028] Optionally, the target is a defect, the first attribute information is a process segment where the defect is located, and the second attribute information is a defect type.

[0029] A second aspect of the present disclosure provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the target classification method described above are implemented.

[0030] A third aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the target classification method described above when the program is executed by a processor.

[0031] A fourth aspect of the present disclosure provides a computer program product, including a computer program, which implements the steps of the target classification method described above when executed by a processor.

[0032] The beneficial effects of the present disclosure are as follows:

[0033] The target classification method of the disclosed embodiment divides the attribute information representing the target into two parts, namely the first attribute information and the second attribute information, and pre-sets the first classification branch and the second classification branch. When performing target classification, the image to be detected or the first area image is input into the first classification branch, and the first target classification (i.e., first attribute classification) is performed through the first classification branch. At the same time, the first area image is input into the second classification branch, and the second target classification (i.e., second attribute classification) is performed through the second classification branch. By separately detecting the first attribute and the second attribute, on the one hand, the types of targets in each classification branch can be greatly reduced, thereby reducing the difficulty of classification. On the other hand, when the second attribute is detected separately, only the local image covering the second attribute, i.e., the first area image, can be used. Compared with the image to be detected when the first attribute and the second attribute are jointly detected, the first area image removes more background information, thereby avoiding interference of background information on the classification process, thereby further improving the accuracy of target classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The specific implementation methods of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0035] Figure 1 A flow chart of a target classification method provided by an embodiment of the present disclosure;

[0036] Figure 2 A schematic diagram of images to be detected corresponding to different process stages provided in an embodiment of the present disclosure;

[0037] Figure 3 A schematic diagram of an image to be detected and an image of a first region provided in an embodiment of the present disclosure;

[0038] Figure 4 A schematic diagram of a network structure of a target classification model provided in one embodiment of the present disclosure;

[0039] Figure 5 A schematic diagram of the network structure of the first target classification model when the first target classification model is a ResNeXt convolutional neural network model;

[0040] Figure 6 A schematic diagram of the network structure of a target classification model provided in another embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0042] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] In order to better understand the technical solutions of the target classification method, terminal, storage medium and computer program product disclosed in the present invention, the design concept of the present invention is first briefly introduced.

[0044] At present, with the increasing application of machine learning and computer vision, machine learning and computer vision technologies can be used in production lines for automatic defect detection and classification to improve detection speed and quality. When performing automatic defect detection on a production line, two pieces of information are usually required: 1. Which process the defect comes from, and 2. The category of the defect itself. A defect is characterized by the above two pieces of information. For example, a defect is represented as PD, where P represents the process segment where the defect is located, and D represents the category of the defect itself.

[0045] In the related art, each defect in the production line is usually regarded as a category to obtain all the defect types to be classified, and then a corresponding classification model is designed. The classification model is then used to classify the defects of the inspection image and output the defect classification results. For example: in a certain production line, there are 8 process sections where defects may occur, which are recorded as process1, process2...process8, and there are 10 possible defect categories, which are recorded as defect1, defect2...defect10. At this time, the corresponding defect types are at most 80. In the actual production line, since some defect categories (defects) will not appear in some process sections (processes), the actual number of defect types may be less than 80, for example, 40, which are recorded as PD1, PD2, PD3...PD40. In the relevant defect classification method, these 40 defects are used as classification results, and the classification model is trained through sample images. After the trained classification model is obtained, the image to be detected is input into the trained classification model, and the defect classification result corresponding to the image to be detected can be directly output. The defect classification result includes the confidence corresponding to each of the 40 defects, and the defects with high confidence are regarded as the defects existing in the image to be detected.

[0046] However, the applicant has found through research that there are the following problems when classifying defects using the above method:

[0047] (1) There are many types of defects, and the number of samples in each type is small, which will make the classification model complex and difficult, and will also affect the classification accuracy of the classification model;

[0048] (2) The process section where the defect is located and the defect category are detected together. Therefore, the image to be detected needs to cover the process information and the defect category. At this time, the image to be detected will contain more background information, which will interfere with the classification process and further affect the classification accuracy.

[0049] In order to solve the above technical problems, the embodiments of the present disclosure provide a target classification method, a terminal, a storage medium and a computer program. The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0050] Please refer to Figure 1 , Figure 1 Flow chart of the target classification method provided by the embodiment of the present disclosure. Figure 1 As shown, the following steps are included:

[0051] Step S101 : acquiring a first region image in an image to be detected, wherein the image to be detected includes first attribute information and second attribute information characterizing a target in the object to be detected, and the first region is a region where the target is located.

[0052] In one possible implementation, the target is a defect, the first attribute information is the process section where the defect is located, the second attribute information is the defect type, the first area is the area where the defect is located, and the first area image is the image of the first area in the image to be detected. It is understandable that when the target classification method of the embodiment of the present disclosure is applied to other scenarios, the target, the first attribute information, and the second attribute information can also be other content, and are not limited to the defects in the embodiment of the present disclosure. Exemplarily, when the target classification method of the embodiment of the present disclosure is used to classify people, the target is a person, the first attribute information can be the gender of the person, and the second attribute information can be the age group of the person, that is, the person is classified by the two characteristics of gender and age group.

[0053] In the following description, the target is defect PD, and the defect PD is divided into two parts, respectively recorded as the first defect and the second defect. The first defect represents the process segment (process) where the defect is located, and the second defect represents the defect type (defect). For example, at this time, the first attribute information represents the first defect, that is, the process segment where the defect is located, and the second attribute information represents the second defect, that is, the defect type.

[0054] Specifically, when the target is a defect, the first attribute information is the process segment where the defect is located, and the second attribute information is the defect type, the image to be tested includes image information characterizing the process segment where the defect is located in the part to be tested and the defect type. Taking the display panel production line as an example, the part to be tested refers to the products output from different process segments in the display panel production line. Exemplarily, the process segment can be characterized by circuits, solder joints, traces and other features in the image to be tested. Different process segments will contain different circuits, solder joints, traces and other features, and these features will cause the image to be tested corresponding to the process segment to present different colors and textures. Therefore, the images to be tested corresponding to different process segments have large overall differences. By performing image processing on the image to be tested, the process segment corresponding to the image to be tested can be identified. For example, please refer to Figure 2 , Figure 2 It is a schematic diagram of the images to be detected corresponding to different process sections. Figure 2 The first image and the second image correspond to the images to be inspected of different process sections, respectively, and the filled area in the first image represents the defect A in the corresponding process section, and the area Q1 represents the image of the first area where the defect A is located, and the filled area in the second image represents the defect B in the corresponding process section, and the area Q2 represents the image of the first area where the defect B is located. It can be understood that for production lines of other products, their different process sections may also be characterized by other features, such as structure, color, shape and other features, rather than being limited to features such as circuits, solder joints, and routing.

[0055] Optionally, in order to detect the process section where the defect is located, the image to be detected is a full image taken of the part to be tested (such as a full image including 1024*1024 pixels), that is, the image to be detected includes all the information of the part to be tested, wherein the information of the part to be tested includes circuits, solder joints, routing and other features. In specific implementation, the image to be detected can be obtained by photographing the products output from each process section on the product production line with a camera set on the product production line.

[0056] In the disclosed embodiment, the first area is the area where the target (such as a defect) is located, and the first area image is the image of the first area in the image to be detected, which can be understood as: the first area image is a local image of the image to be detected. Among them, different types of defects have different morphology, shape, position and other characteristics. For example, the defect is in the shape of a point, line, circle, etc., which can be characterized by a local image of the location of the defect (for example, the local image includes 20*10 pixels). For any process section, the types of defects that may occur in the process section are fixed. For example, there may be 3 types of defects in a process section P1, which are represented as D1, D2, and D3, respectively, and there may be 4 types of defects in the process section P2, which are represented as D4, D5, D6, and D7, respectively, and the positions of the defects D1, D2, D3, D4, D5, D6, and D7 in the product are usually determined and different, so the position of the defect can be set as the first area. It can be understood that the position of the first area in the image to be detected can be obtained based on experience.

[0057] For example, please refer to Figure 3 , Figure 3 is a schematic diagram of an embodiment of an image to be detected and a first area image, such as Figure 3 As shown, the image to be detected Io is a full image obtained by photographing the object to be detected, and the first area image Is is a local image including defects in the image to be detected Io.

[0058] Step S102: input the first area image or the image to be detected into a first classification branch, and input the first area image into a second classification branch, wherein the first classification branch is configured to perform a first target classification on the input first area image or the image to be detected and output a first target classification result, and the second classification branch is configured to perform a second target classification on the input first area image and output a second target classification result, wherein the first target classification result represents first attribute information, and the second target classification result represents second attribute information.

[0059] In the disclosed embodiment, a target is characterized by a first attribute and a second attribute. When classifying the target in the image to be detected, two classification branches (i.e., a first classification branch and a second classification branch) are designed. The first classification branch is used to detect and classify the first target (i.e., the first attribute) of the test piece, and the second classification branch is used to detect and classify the second target (i.e., the second attribute) of the test piece.

[0060] Among them, the first classification branch and the second classification branch constitute a target classification model (exemplarily, the target classification model is a defect classification model), and the target classification model adopts a deep learning (Deep Learning, DL) network model. Exemplarily, the target classification model is a convolutional neural network model or a densely connected convolutional network model (Densely Connected Convolutional Network, referred to as DenseNet), wherein the convolutional neural network model can adopt network structures such as Inception-ResNet, ResNet, and ResNeXt. It is understandable that the target classification model can also adopt other mature deep learning network models in related technologies.

[0061] In a possible implementation, the target classification model includes a first classification branch and a second classification branch. The inputs of the first classification branch and the second classification branch are both the first region image Is, the first classification branch is used to perform a first target classification on the input first region image Is and output the first target classification result, and the second classification branch is used to perform a second target classification on the input first region image Is and output the second target classification result.

[0062] Exemplarily, when the target is a defect, the first attribute information represents the first defect, and the second attribute information represents the second defect, since the first area image Is is an image of the area where the defect is located in the image to be detected Io, the image mainly carries image information characterizing the defect type. On the one hand, since the defect type and the process segment have a certain correspondence (such as the defect type and the process segment are a one-to-one or one-to-many correspondence), the image information characterizing the defect type can also characterize the process segment where the defect is located to a certain extent. On the other hand, the first area image Is is a local image, so it will also carry a small amount of image information characterizing the process segment where the defect is located. Therefore, by performing detection and classification of the first defect on the first area image Is through the first classification branch, the classification result of the first defect (i.e., the first defect classification result) can be obtained, and by performing detection and classification of the second defect on the first area image Is through the second classification branch, the classification result of the second defect can be obtained.

[0063] In one possible implementation, the input of the first classification branch is the image to be detected Io, and the input of the second classification branch is the first area image Is. The first classification branch is used to perform a first target classification on the input image to be detected Io and output the first target classification result, and the second classification branch is used to perform a second target classification on the input first area image Is and output the second target classification result.

[0064] Step S103: generating a third target classification result of the image to be detected according to the first target classification result and the second target classification result, wherein the third target classification result includes the first target classification result and the second target classification result.

[0065] Compared with the related art, the target classification method of the embodiment of the present invention divides the attribute information representing the target into two parts, namely the first attribute information and the second attribute information, and pre-sets the first classification branch and the second classification branch. When performing target classification, the image to be detected or the first area image is input into the first classification branch, and the first target classification (i.e., first attribute classification) is performed through the first classification branch. At the same time, the first area image is input into the second classification branch, and the second target classification (i.e., second attribute classification) is performed through the second classification branch. By separately detecting the first attribute and the second attribute, on the one hand, the types of targets in each classification branch can be greatly reduced, thereby reducing the difficulty of classification. On the other hand, when the second attribute is detected separately, only the local image covering the second attribute, i.e., the first area image, can be used. Compared with the image to be detected when the first attribute and the second attribute are jointly detected, the first area image removes more background information, thereby avoiding interference of background information on the classification process, thereby further improving the accuracy of target classification.

[0066] Exemplarily, when the target is a defect, the first attribute information represents the first defect, and the second attribute information represents the second defect, the target classification method of the embodiment of the present disclosure divides the defect of the test piece into two parts, represented as the first defect and the second defect, and pre-sets the first classification branch and the second classification branch. When classifying the defects of the test piece, the image to be detected or the first area image is input into the first classification branch, and the first defect classification is performed through the first classification branch. At the same time, the first area image is input into the second classification branch, and the second defect classification is performed through the second classification branch. By separately detecting the first defect and the second defect, on the one hand, the types of defects in each classification branch can be greatly reduced, thereby reducing the difficulty of classification. On the other hand, when the second defect is detected separately, only the local image covering the defect category, that is, the first area image, can be used. Compared with the image to be detected when the first defect and the second defect are jointly detected, the first area image removes more background information, thereby avoiding interference of background information on the classification process, thereby further improving the accuracy of defect classification.

[0067] The target classification method of the embodiment of the present disclosure performs first defect classification through the first classification branch and performs second defect classification through the second classification branch, thereby realizing the separate detection of the first defect and the second defect. Since the first defect and the second defect are detected separately, the number of samples under each classification is increased, thereby reducing the difficulty of classification and improving the classification accuracy. For example, assuming that there are 1,000 sample images, the first defects include 8 types, the second defects include 10 types, and there are 40 defect classifications. When the existing defect detection method in the related art is used for detection, the number of samples under each classification may be only 25. However, through the target classification method of this embodiment, in the first defect classification process, the number of samples corresponding to each first defect may be 125, and the number of samples corresponding to each second defect may be 100, that is, the number of samples corresponding to each defect category is greatly increased.

[0068] In a possible implementation, the first classification branch and the second classification branch are partially decoupled. Partial decoupling means that the classification process of the first attribute and the classification process of the second attribute have a certain dependency relationship, that is, the first classification branch and the second classification branch have a part of the common network structure. Figure 4 , Figure 4 A schematic diagram of an embodiment of a target classification model provided in an embodiment of the present disclosure, for Figure 4 The technical solution of partially decoupling the first attribute and the second attribute shown in the figure, the target classification model is represented as a first target classification model, specifically, the first target classification model includes an image processing module 10, a first classification module 201 and a second classification module 301, the image processing module 10 and the first classification module 201 form a first classification branch, the image processing module 10 and the second classification module 301 form a second classification branch, the image processing module 10 includes an input end, a first output end and a second output end, the input end is configured to input the first region image Is, the first output end is connected to the input end of the first classification module 201, and the second The output end is connected to the input end of the second classification module 301, the image processing module 10 is configured to perform convolution and down-sampling processing on the first area image Is to obtain a second area image, and output the second area image to the first classification module 201 through the first output end, and output the second area image to the second classification module 301 through the second output end, the first classification module 201 is configured to perform a first target classification on the input second area image and output a first target classification result, and the second classification module 301 is configured to perform a second target classification on the input second area image and output a second target classification result.

[0069] In the above embodiment, the backbone network of the first target classification model is divided into two parts, the first part is the image processing module 10, and the second part is the first classification module 201 and the second classification module 301, wherein the image processing module 10 can be understood as the common image processing part of the first target classification and the second target classification, and the second part can be understood as the image processing part of the first target classification and the second target classification. Since the first target classification and the second target classification have a part of the common network structure (i.e., the image processing module 10), there is a certain dependency between the classification processes of the first target classification and the second target classification, and the two are partially decoupled.

[0070] It is understandable that before using the target classification method of the embodiment of the present disclosure to classify the target image to be detected, it is necessary to pre-train the target classification model. Figure 4 Before the step of obtaining the first region image in the image to be detected, the target classification method further comprises: training the initial first target classification model to obtain a trained first target classification model, wherein the training process comprises the following steps:

[0071] Step S201, obtaining a first training set, wherein the first training set includes a plurality of first sample images, wherein the first sample image is an image at a second region in an initial sample image, wherein the initial sample image includes first attribute information and second attribute information characterizing a target in a sample piece, and the second region is a region where the target is located.

[0072] Exemplarily, when the target is a defect, the initial sample image is a full image taken of the sample part, and the initial sample image includes image information characterizing the process section and defect type of the defect in the sample part. The first sample image is a local image of the defect area obtained by intercepting the initial sample image.

[0073] When the first sample image is used as the first training set to train the initial first target classification model, the first sample image needs to be manually marked in advance with the true value (Groud Truth) of the target classification result. The true value includes the first target true value (Process Type) and the second target true value (Defect Type). The first target true value represents the true value of the first attribute in the first sample image (such as the process section where the defect is located), and the second target true value represents the true value of the second attribute in the first sample image (such as the defect type).

[0074] Step S202, training the initial first target classification model through the first training set to obtain a trained first target classification model, wherein the first target classification model includes the first classification branch and the second classification branch, and the first classification branch is configured to perform a first target classification on the input first area image and output a first target classification result, and the second classification branch is configured to perform a second target classification on the input first area image and output a second target classification result.

[0075] Optionally, the step of training the initial first target classification model by using the training set to obtain the trained first target classification model includes:

[0076] (1) Inputting a first sample image into an initial first target classification model to obtain a first target prediction value and a second target prediction value, wherein the first target prediction value represents a prediction value of a first attribute in the first sample image, and the second target prediction value represents a prediction value of a second attribute in the first sample image, and the network parameters in the initial first target classification model have initial values.

[0077] Specifically, the first target classification model is a convolutional neural network model, and the classification ability of the classification model is generally measured by a loss function. Generally, the smaller the value of the loss function, the closer the probability predicted by the model is to the true distribution, that is, the trained model has a better classification ability for the input sample. In the disclosed embodiment, during the training process of the first target classification model, the network parameters of each layer are updated by a back propagation algorithm until the loss function converges or is less than a set value. Before training, the network parameters in the initial first target classification model have initial values, and the network parameters include the weights w and biases b of each layer. During the training process, the network parameters of each layer are updated by a back propagation algorithm.

[0078] (2) Obtain a first target true value and a second target true value of the first sample image.

[0079] (3) Calculate a first loss value based on the first target true value and the first target predicted value, calculate a second loss value based on the second target true value and the second target predicted value, and perform weighted summation of the first loss value and the second loss value according to a preset weight coefficient to obtain a third loss value, wherein the preset weight coefficient includes a first weight coefficient corresponding to the first loss value and a second weight coefficient corresponding to the second loss value.

[0080] Exemplarily, the third loss value L=w p *L p +w d *L d , where L pRepresents the first loss value, L d represents the second loss value, w p Represents the first weight coefficient corresponding to the first loss value, w d Represents the second weight coefficient corresponding to the second loss value.

[0081] (4) updating the network parameters in the initial first target classification model according to the third loss value and the back propagation algorithm until the third loss value converges.

[0082] It should be noted that the back propagation algorithm may adopt existing mature technologies, and the embodiments of the present disclosure do not limit the specific back propagation algorithm.

[0083] In a possible implementation, the values ​​of the first weight coefficient and the second weight coefficient are positively correlated with the classification difficulty of the first attribute and the second attribute, that is, the greater the classification difficulty, the greater the corresponding weight coefficient.

[0084] Optionally, the classification difficulty can be characterized by the number of defect types. Generally, the more defect types there are, the greater the classification difficulty is, and the fewer defect types there are, the smaller the classification difficulty is. Therefore, in one implementation, the first weight coefficient w can be set p is the number of categories of the first attribute, and the second weight coefficient w d is the number of categories of the second attribute. Assume that the number of categories of the first attribute is represented by C p , the number of categories of the second attribute is denoted as C d , then w p =C p , w d =C d .

[0085] In addition, the classification difficulty can also be represented by the accuracy of the classification result. The higher the accuracy of the classification result, the smaller the classification difficulty, and the lower the accuracy of the classification result, the greater the classification difficulty. At this time, the values ​​of the first weight coefficient and the second weight coefficient can be set to be inversely proportional to the classification accuracy of the first attribute and the second attribute, such as the first weight coefficient w p is the accuracy of the second target classification result in the first target classification model, and the second weight coefficient w d is the accuracy of the first target classification result in the first target classification model. Assume that the accuracy of the first target classification result is expressed as A p , the accuracy of the second target classification result is expressed as A d , you can set w p =A d , w d =A p .

[0086] In the disclosed embodiment, in addition to setting a training set, a validation set is also set during the training process of the first target classification model. The training set is mainly used to train and adjust model parameters, while the validation set is used to verify model accuracy and adjust model hyperparameters. For example, by checking the relationship between the loss values ​​of the training set and the validation set as the epoch changes, it can be seen whether the model is overfitting. If so, the training can be stopped in time, and then the model structure and hyperparameters can be adjusted according to the situation, which greatly saves time. An epoch is the process of training all training samples once, that is, all training samples undergo one forward propagation and one backward propagation in the neural network. The accuracy of the first target classification and the second target classification on the validation set after each epoch can be counted as the accuracy of the first target classification result A and B respectively. p , the accuracy of the second target classification result A d .

[0087] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram of the network structure of the first target classification model when the first target classification model is a ResNeXt convolutional neural network model. Figure 5 As shown, the image processing module 10 includes an input layer 101, a convolution layer 102, a first pooling layer 103, a first group convolution block 104, and a second group convolution block 105 connected in sequence. The first classification module 201 and the second classification module 301 have the same network structure, and respectively include a third group convolution block 106, a fourth group convolution block 107, a second pooling layer 108, and a fully connected layer 109 connected in sequence. The output end of the second group convolution block 105 is connected to the input end of the third group convolution block 106. The fully connected layer of the first classification module 201 outputs the first target classification result, and the fully connected layer of the second classification module 301 outputs the second target classification result. It can be understood that although the network structures of the first classification module 201 and the second classification module 301 are the same, their specific network parameters are different, and the specific network parameters are updated during the training process.

[0088] Among them, the input layer, as the first layer of the convolutional neural network, represents the pixel matrix of the input image, namely, length, width, and height. The length and width represent the size of the image; the height represents the color channel of the image, the number of channels for black and white images is 1, and the number of channels in RGB mode is 3. Its main function is to normalize the input data. The convolution layer is the core operation of the convolutional neural network. It is composed of multiple convolution templates. The parameters (weights and biases) of the convolution templates in the convolutional neural network are shared, that is, the convolution operation for the same feature map channel uses the same convolution template parameters. During the convolution operation, the convolution template performs convolution multiplication operations with each pixel in the receptive field one by one, and obtains the feature information of the image through a nonlinear activation function. Exemplarily, the size of the convolution template is 3*3 and 5*5. The pooling layer (Pooling Layer), also known as the downsampling layer, can reduce the dimension of the feature map after convolution, compress the image resolution, and reduce redundant information, thereby reducing the amount of calculation that is greatly increased due to the increase in the number of network layers. The pooling operation usually uses a pooling window of size 2*2, and traverses the entire image with a fixed step size (usually set to 2). Pooling is divided into maximum pooling and average pooling. The maximum pooling operation can obtain the maximum value of the pixel points in the local range covered by the pooling window, and can better retain the edge features and texture features of the original image. Therefore, in the embodiment disclosed in the present invention, the first pooling layer adopts the maximum pooling layer. The group convolution block (ResNeXt Block) is the core structure of the ResNeXt convolutional neural network model. It improves the performance of the model by using group convolution (Group Convolution). Group convolution is a convolution operation that divides the input feature map into several groups, and then uses a convolution kernel (filter) for each group. The full connection layer (Full Connection Layer), located at the back end of the network, uses a fully connected method to connect the neurons of the front and back two layers, and can be used as a classifier. Its task is to tile the feature map obtained by the last convolution pooling layer to obtain a one-dimensional feature vector, and calculate the vector to obtain the probability that the original image belongs to each category, and finally establish a mapping relationship from feature space to category space.

[0089] In one possible implementation, the classification of the first attribute and the second attribute are completely decoupled. The complete decoupling in the embodiment of the present disclosure means that the classification process of the first attribute and the classification process of the second attribute are implemented completely independently without any dependency on each other, that is, the first classification branch and the second classification branch are independent of each other and there is no common network structure.

[0090] For details, please refer to Figure 6 , Figure 6 This is a schematic diagram of the network structure of the target classification model provided by the embodiment of the present disclosure. Figure 6As shown, at this time, the target classification model includes a first classification branch 20 and a second classification branch 30, and the first classification branch 20 and the second classification branch 30 are independently trained through their respective training sets. At this time, before the step of obtaining the first area image in the image to be detected, it also includes:

[0091] Step S301, obtaining a second training set and a third training set, wherein the second training set includes a plurality of second sample images, and the third training set includes a plurality of third sample images, wherein the second sample images include first attribute information and second attribute information characterizing a target in a sample piece, and the third sample images are images of a third area in the second sample images, and the third area is an area where the target is located.

[0092] Step S302, training the initial second target classification model through the second training set to obtain a trained second target classification model, the second target classification model serves as the first classification branch, and the first classification branch is configured to perform a first target classification on the input image to be detected and output a first target classification result.

[0093] Step S303: train the initial third target classification model using the third training set to obtain a trained third target classification model, and use the third target classification model as the second classification branch.

[0094] Exemplarily, the initial second target classification model and the initial third target classification model are convolutional neural network models; the step of training the initial second target classification model through the second training set to obtain the trained second target classification model includes: inputting the second sample image into the initial second target classification model to obtain a first target prediction value; obtaining a first target true value of the second sample image, and calculating a first loss value according to the first target true value and the first target prediction value; updating the network parameters in the initial second target classification model according to the first loss value and the back propagation algorithm until the first loss value reaches a preset stop condition;

[0095] The initial third target classification model is trained by the third training set to obtain a trained third target classification model, including the steps of: inputting the third sample image into the initial third target classification model to obtain a second target prediction value; obtaining the first target true value of the third sample image, and calculating a second loss value based on the first target true value and the second target prediction value; and updating the network parameters in the initial third target classification model based on the second loss value and the back propagation algorithm until the second loss value reaches a preset stop condition.

[0096] It is understandable that the network structures of the second target classification model and the third target classification model in the embodiment of the present disclosure may be the same or different, that is, the two are completely independent. Further, the second target classification model and the third target classification model are ResNeXt convolutional neural network models or DenseNet convolutional neural network models. Among them, the training process principle of the second target classification model and the third target classification model is the same as that of the first target classification model, and will not be described in detail here.

[0097] Based on the same inventive concept, the second aspect of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the target classification method as described above when executing the program.

[0098] Based on the same inventive concept, the third aspect of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the target classification method described above are implemented.

[0099] In the specific implementation process, the computer storage medium may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other storage media that can store program codes.

[0100] Based on the same inventive concept, the fifth aspect of the present disclosure provides a computer program product, including a computer program, which implements the steps of the target classification method described above when executed by a processor. Since the principle of solving the problem by the above computer program is similar to the principle of the target classification method, the implementation of the above computer program can refer to the implementation of the target classification method, and the repeated parts will not be repeated.

[0101] The computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0102] Obviously, the above embodiments of the present disclosure are merely examples for clearly illustrating the present disclosure, and are not limitations on the implementation methods of the present disclosure. For ordinary technicians in this field, other different forms of changes or modifications can be made on the basis of the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present disclosure are still within the protection scope of the present disclosure.

Claims

1. A target classification method, characterized in that: The following steps are involved: Acquire a first area image in an image to be detected, wherein the image to be detected includes first attribute information and second attribute information characterizing a target in the object to be detected, and the first area is an area where the target is located; Inputting the first region image or the image to be detected into a first classification branch, and inputting the first region image into a second classification branch, wherein the first classification branch is configured to perform a first target classification on the input first region image or the image to be detected and output a first target classification result, and the second classification branch is configured to perform a second target classification on the input first region image and output a second target classification result, wherein the first target classification result represents the first attribute information, and the second target classification result represents the second attribute information; A third target classification result of the image to be detected is generated according to the first target classification result and the second target classification result.

2. The target classification method according to claim 1, characterized in that: Before the step of acquiring the first area image in the image to be detected, the method further includes: Acquire a first training set, wherein the first training set includes a plurality of first sample images, wherein the first sample image is an image at a second region in an initial sample image, wherein the initial sample image includes first attribute information and second attribute information characterizing an object in a sample, and the second region is a region where the object is located; The initial first target classification model is trained by using the first training set to obtain a trained first target classification model, wherein the first target classification model includes the first classification branch and the second classification branch, and the first classification branch is configured to perform a first target classification on the input first area image and output a first target classification result.

3. The target classification method according to claim 2, characterized in that: The first target classification model includes an image processing module, a first classification module and a second classification module. The image processing module and the first classification module form the first classification branch, and the image processing module and the second classification module form the second classification branch. The image processing module includes an input end, a first output end and a second output end. The input end is configured to input the first area image, the first output end is connected to the input end of the first classification branch, and the second output end is connected to the input end of the second classification branch. The image processing module is configured to perform convolution and downsampling processing on the first area image to obtain a second area image, and output the second area image to the first classification module through the first output end, and output the second area image to the second classification module through the second output end. The first classification module is configured to perform a first target classification on the input second area image and output a first target classification result, and the second classification module is configured to perform a second target classification on the input second area image and output a second target classification result.

4. The target classification method according to claim 3, characterized in that: The initial first target classification model is a convolutional neural network model; the step of training the initial first target classification model by using the training set to obtain the trained first target classification model comprises: Inputting the first sample image into the initial first target classification model to obtain a first target prediction value and a second target prediction value, wherein the first target prediction value represents a prediction value of a first attribute in the first sample image, and the second target prediction value represents a prediction value of a second attribute in the first sample image, and the network parameters in the initial first target classification model have initial values; Acquire a first target true value and a second target true value of the first sample image, wherein the first target true value represents a true value of a first attribute in the first sample image, and the second target true value represents a true value of a second attribute in the first sample image; Calculating a first loss value according to the first target true value and the first target predicted value, calculating a second loss value according to the second target true value and the second target predicted value, and performing weighted summation of the first loss value and the second loss value according to a preset weight coefficient to obtain a third loss value; The network parameters in the initial first target classification model are updated according to the third loss value and the back propagation algorithm until the third loss value converges.

5. The target classification method according to claim 4, characterized in that: The preset weight coefficients include a first weight coefficient corresponding to the first loss value and a second weight coefficient corresponding to the second loss value, and the values ​​of the first weight coefficient and the second weight coefficient are positively correlated with the classification difficulty of the first attribute and the second attribute.

6. The target classification method according to claim 5, characterized in that: The first weight coefficient is the number of categories of the first attribute, and the second weight coefficient is the number of categories of the second attribute; or the first weight coefficient is the accuracy of the second target classification result in the first target classification model, and the second weight coefficient is the accuracy of the first target classification result in the first target classification model.

7. The target classification method according to claim 3, characterized in that: The first target classification model is a ResNeXt convolutional neural network model, the image processing module includes an input layer, a convolution layer, a first pooling layer, a first group convolution block and a second group convolution block connected in sequence, the first classification module and the second classification module have the same network structure, respectively including a third group convolution block, a fourth group convolution block, a second pooling layer and a fully connected layer connected in sequence, the output end of the second group convolution block is connected to the input end of the third group convolution block, the fully connected layer of the first classification module outputs the first target classification result, and the fully connected layer of the second classification module outputs the second target classification result.

8. The target classification method according to claim 1, characterized in that: Before the step of acquiring the first area image in the image to be detected, the method further includes: Acquire a second training set and a third training set, wherein the second training set includes a plurality of second sample images, and the third training set includes a plurality of third sample images, wherein the second sample images include first attribute information and second attribute information characterizing an object in a sample piece, and the third sample images are images of a third region in the second sample images, and the third region is a region where the object is located; The initial second target classification model is trained by the second training set to obtain a trained second target classification model, the second target classification model is used as the first classification branch, and the first classification branch is configured to perform a first target classification on the input image to be detected and output a first target classification result; The initial third target classification model is trained using the third training set to obtain a trained third target classification model, and the third target classification model serves as the second classification branch.

9. The target classification method according to claim 8, characterized in that: The second target classification model and the third target classification model are ResNeXt convolutional neural network models or DenseNet convolutional neural network models.

10. The target classification method according to any one of claims 1 to 9, characterized in that: The target is a defect, the first attribute information is a process section where the defect is located, and the second attribute information is a defect type.

11. A terminal, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the target classification method as described in any one of claims 1 to 10 when executing the program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the target classification method according to any one of claims 1 to 10 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target classification method according to any one of claims 1 to 10 are implemented.

Citation Information

Patent Citations

  • Target recognition model training method and device and electronic equipment

    CN112990432A

  • Vehicle classification method and device, computer equipment and storage medium

    CN116188832A

  • Defect detection method, electronic equipment and storage medium

    CN116579986A

  • Panel detection method, device and equipment and storage medium

    CN116735598A