Material defect information identification method, model training method and equipment

Abstract

The embodiment of the invention provides a material defect information identification method, a model training method and equipment. The method comprises the steps that to-be-recognized data is acquired, and the to-be-recognized data comprises a to-be-recognized material defect image and at least one kind of scene data; then, feature extraction is carried out on a to-be-recognized material defect image according to the recognition model, and implicit features of the to-be-recognized material defect image are obtained; afterwards, according to a scene learner corresponding to the scene data in the recognition model, feature extraction is carried out on the scene data, and scene features of the scene data are obtained; and determining defect information of the to-be-recognized material defect image based on the recognition model, the scene features and the implicit features. The method is used for achieving the effect of accurately and effectively detecting the defect information of the material under different scene data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a material defect information recognition method, model training method and equipment. Background Art

[0002] During the manufacturing process of materials (such as wafers and other materials used to manufacture semiconductors), it is necessary to detect defect information of the materials, wherein the defect information includes, but is not limited to, one of defect equivalent size or defect type.

[0003] To identify defect information, the defect image and initial defect type of the material can be detected by hardware equipment, and image features of the defect image can be extracted based on a predefined algorithm in the hardware equipment to obtain an image attribute of the defect image, and the defect information can be determined based on a preset relationship between the image attribute and the defect information in the defect image.

[0004] However, in the above method, changes in scene data will affect the relationship between image attributes and defect information in the defect image. Therefore, if the scene data of the material detected by the hardware device changes, the predefined algorithm in the hardware device needs to be modified to more accurately determine the defect information. The operation is complicated, the prediction speed is slow, and the efficiency is low. Summary of the invention

[0005] The embodiments of the present application provide a material defect information identification method, a model training method and a device, so as to achieve the effect of accurately and effectively detecting the defect information of materials under different scene data.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying defect information of a material, comprising:

[0007] Acquire data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data;

[0008] Extracting features of the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified; wherein the recognition model includes at least one scene learner;

[0009] Extracting features of the scene data according to a scene learner corresponding to the scene data in the recognition model to obtain scene features of the scene data;

[0010] Based on the recognition model, the scene features and the implicit features, the defect information of the material defect image to be identified is determined.

[0011] The material defect information identification method provided in the embodiment of the present application includes at least one scene learner in the recognition model for acquiring features of the material defect image to be identified, and then the scene data can be input into the scene learner corresponding to the scene data in the recognition model according to the recognition model for feature extraction, and the scene features of the scene data can be obtained, and then the defect information can be identified based on the scene features and implicit features. Therefore, it can simultaneously support accurate defect identification of the data to be identified containing multiple scene data, and the operation is simple, the recognition speed is fast, and the efficiency is high.

[0012] In a possible implementation manner, the scene feature is used to characterize the relationship between the defect in the material defect image to be identified and the scene data; and based on the recognition model, the scene feature and the implicit feature, the defect information of the material defect image to be identified is determined, including:

[0013] Correcting the latent feature according to the scene feature to obtain a corrected latent feature;

[0014] Based on the recognition model and the corrected implicit features, defect information of the material defect image to be recognized is determined.

[0015] By correcting the implicit features based on the scene features, the corrected implicit features are obtained, which makes the corrected implicit features more consistent with the variation law of the defect features in the material defect image to be identified with the scene data, so the corrected implicit features are more accurate. Then, based on the recognition model and the more accurate corrected implicit features, more accurate defect information of the material defect image to be identified can be determined.

[0016] In a possible implementation manner, determining defect information of the material defect image to be identified based on the recognition model and the corrected implicit feature includes:

[0017] Performing image feature extraction on the material defect image to be identified based on an image processing method to obtain physical features of the material defect image to be identified; wherein the physical features of the material defect image to be identified represent at least one physical attribute of the defect in the material defect image to be identified;

[0018] The physical feature is fused with the corrected implicit feature to obtain a fused feature; and the fused feature is identified based on the identification model to obtain defect information of the material defect image to be identified.

[0019] In a possible implementation, the recognition model includes a plurality of task recognition units, and the data to be recognized indicates an identifier of defect information to be recognized; and the fusion feature is recognized based on the recognition model to obtain defect information of the material defect image to be recognized, including:

[0020] The fusion feature is input into the task identification unit corresponding to the identifier of the defect information to be identified in the identification model for processing, so as to obtain the defect information of the material defect image to be identified.

[0021] In a possible implementation, the recognition model includes a deep neural network unit; and extracting features of the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified includes:

[0022] Based on the deep neural network unit in the recognition model, feature extraction is performed on the material defect image to be identified to obtain implicit features of the material defect image to be identified; wherein the implicit features of the material defect image to be identified represent key information of the defects in the material defect image to be identified.

[0023] In a second aspect, an embodiment of the present application provides a model training method for defect information recognition, including:

[0024] Acquire data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data;

[0025] Extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained; wherein the initial model includes at least one scene learner; and extracting features of the scene data according to the scene learner corresponding to the scene data in the initial model to obtain scene features of the scene data;

[0026] A recognition model is determined according to the initial model, the implicit features and the scene features; wherein the recognition model is used to identify the material defect image to be identified to obtain defect information.

[0027] The defect information model training method for materials provided in the embodiment of the present application includes at least one scene learner in the initial model for feature extraction of the training data, and then the scene data can be input into the scene learner corresponding to the scene data in the initial model for feature extraction according to the initial model, and the scene features of the scene data can be obtained, and then the initial model can be trained based on the scene features and implicit features. Therefore, it can simultaneously support model training of the training data containing multiple scene data, with simple operation, fast training speed and high efficiency.

[0028] In a possible implementation manner, the scene feature represents the relationship between the defect in the material defect image to be trained and the scene data; and determining the recognition model according to the initial model, the implicit feature and the scene feature includes:

[0029] Correcting the latent feature according to the scene feature to obtain a corrected latent feature;

[0030] The recognition model is determined according to the initial model and the modified implicit features.

[0031] By correcting the implicit features based on the scene features, the corrected implicit features are obtained, which makes the corrected implicit features more consistent with the variation law of the defect features in the material defect image to be trained with the scene data. Therefore, the corrected implicit features are more accurate. Then, based on the initial model and the more accurate corrected implicit features, a recognition model that can more accurately identify defect information can be determined.

[0032] In a possible implementation manner, determining the recognition model according to the initial model and the modified implicit features includes:

[0033] Performing image feature extraction on the material defect image to be trained based on an image processing method to obtain physical features of the material defect image to be trained; wherein the physical features of the material defect image to be trained represent at least one physical attribute of the defect in the material defect image to be trained;

[0034] The physical feature and the modified implicit feature are subjected to feature fusion to obtain a fusion feature; and the initial model is trained according to the fusion feature to obtain a recognition model.

[0035] In a possible implementation manner, the image feature extraction of the material defect image to be trained based on an image processing method to obtain the physical features of the material defect image to be trained includes:

[0036] Extracting image features of the material defect image to be trained based on an image processing method to obtain initial image features of the material defect image to be trained;

[0037] If the training stage of the initial model is a pre-training stage, the initial image features of the material defect image to be trained are corrected according to a preset adjustment method of the physical features to obtain the physical features of the material defect image to be trained;

[0038] If the training phase of the initial model is a tuning phase, the initial image features of the material defect image to be trained are determined to be physical features of the material defect image to be trained.

[0039] In a possible implementation manner, the initial model includes a plurality of task identification units, and the data to be trained includes at least one calibrated defect information; and the initial model is trained according to the fusion feature to obtain the identification model, including:

[0040] Inputting the fused features into the task identification unit corresponding to the calibrated defect information in the initial model for processing to obtain identified defect information;

[0041] Obtaining a loss value of the task identification unit according to the calibrated defect information corresponding to the task identification unit and the identified defect information corresponding to the task identification unit;

[0042] According to the loss value of the task identification unit, the parameters of the initial model are adjusted to obtain a recognition model.

[0043] In a possible implementation, adjusting parameters of the initial model according to the loss value of the task identification unit to obtain the identification model includes:

[0044] If the training stage of the initial model is a pre-training stage, according to the loss value of the task identification unit, parameters of each unit in the initial model except for the task identification unit that does not correspond to the calibrated defect information and the scene learner that does not correspond to the scene data are adjusted to obtain a pre-training model;

[0045] If the training stage of the initial model is a tuning stage, according to the loss value of the task identification unit, the parameters of each unit in the pre-trained model except the deep neural network unit, the task identification unit that does not correspond to the calibrated defect information, and the scene learner that does not correspond to the scene data are adjusted to obtain a recognition model.

[0046] In a possible implementation, the initial model includes a deep neural network unit; and extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained includes:

[0047] Based on the deep neural network unit in the initial model, feature extraction is performed on the material defect image to be trained to obtain implicit features of the material defect image to be trained; wherein the implicit features of the material defect image to be trained represent key information of defects in the material defect image to be trained.

[0048] In a third aspect, an embodiment of the present application provides a material defect information identification device, including a module or unit for implementing the material defect information identification method provided in the first aspect and / or various possible implementation methods of the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a model training device for defect information identification, including a module or unit for implementing the model training method provided in the second aspect and / or various possible implementation methods of the second aspect.

[0050] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0051] The memory stores computer-executable instructions;

[0052] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementations of the first aspect as described above, or, the processor executes the second aspect and / or various possible implementations of the second aspect as described above.

[0053] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation methods of the first aspect as described above, or, when the computer execution instructions are executed by a processor, they are used to implement the second aspect and / or various possible implementation methods of the second aspect as described above.

[0054] In the seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the first aspect above and / or various possible implementation methods of the first aspect, or, when executed by a processor, implements the second aspect above and / or various possible implementation methods of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a model training method for defect information recognition provided in this application;

[0056] Figure 2 A flowchart of another model training method for defect information recognition provided in this application;

[0057] Figure 3 An architectural diagram of a model for defect information identification provided in this application;

[0058] Figure 4 A schematic diagram of a process for identifying defect information of a material provided in this application;

[0059] Figure 5 A schematic diagram of a process for identifying defect information of another material provided in this application;

[0060] Figure 6 A schematic diagram of the structure of a defect information identification device for the material provided in this application;

[0061] Figure 7 A schematic diagram of the structure of a model training device for defect information recognition provided in this application;

[0062] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] In order to ensure the quality of materials (such as wafers and other materials used to make semiconductors), it is necessary to detect the materials during the material production process to obtain defect information, wherein the defect information includes, but is not limited to, defect equivalent size and / or defect type. The defect type includes, but is not limited to, any one or more of particle contamination, scratches, cracks, etc. The defect equivalent size can be, for example, the diameter of a cylindrical defect with the same scattered light intensity as the defect in the material.

[0066] In one example, the hardware device can first detect the defect image and initial defect type of the material, and then extract image features of the defect image based on the image processing algorithm in the program preinstalled in the hardware device to obtain the grayscale value of the image, and then determine the defect information based on the preset relationship between the grayscale value of the image and the defect information in the defect image.

[0067] However, in the above method, if the scene data of the material detected by the hardware device changes, the relationship between the grayscale value of the image and the defect information in the defect image will also change. At this time, it is necessary to modify the algorithm of the program pre-installed in the hardware device to accurately determine the defect information. The operation is complicated, the prediction speed is slow, and the efficiency is low.

[0068] The present application provides a material defect information identification method, a model training method and a device to solve the above-mentioned problems.

[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] Figure 1 A flow chart of a model training method for defect information recognition provided in this application is shown as follows: Figure 1 As shown, the method includes:

[0071] S101. Acquire data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data.

[0072] Exemplarily, the execution subject of the model training method provided in this application can be a server, a distributed system, a terminal device, other electronic equipment / computer equipment, or any other device or equipment that can implement the solution of this application, without limitation. Among them, the server can be an independent server, or a server cluster, such as but not limited to any form including a cloud server, a distributed server, a blockchain server, etc. In each embodiment of this application, the execution subject is introduced as a server, which will not be repeated below.

[0073] Exemplarily, the data to be trained can be generated in the following manner, for example:

[0074] Obtain multiple defective materials. One or more defects may exist in a material. The layer types of the materials may be the same or different, and may be determined according to requirements, and this embodiment does not limit this. Among them, the layer type refers to the top layer of the material to be defect-detected using hardware equipment, and can also be understood as the layer where the defects of the material are located. For example, the layer types of a wafer include a silicon layer, an oxide layer, a nitride layer, a polysilicon layer, a metal layer, an insulating layer, a passivation layer, and the like. If the top layer of a defective wafer is a metal layer, the layer type of the wafer is a metal layer, and it can also be understood that there are defects in the metal layer of the wafer.

[0075] For each layer type of multiple defective materials, an optical mode is first set on a hardware device (such as an optical microscope, an X-ray microscope, etc.) (for example, the optical mode can be set according to experience or needs, and this embodiment does not limit this), and then the above materials are transported and positioned by using a mechanical arm and a conveyor belt, and then the above materials are inspected for defects by the hardware device (such as an optical microscope, an X-ray microscope, etc.), and multiple defect (patch) images and the initial defect type corresponding to each defect image are obtained. Among them, the defect image refers to a small image block containing one or more defects that is cropped from the original detected image by the hardware device.

[0076] Among them, the defect image can be used to indicate the defects of the material. The meaning of the defect type can be found in the above introduction and will not be repeated here. The initial defect types corresponding to the defect images can be the same or different, which can be determined according to the needs, and the present embodiment does not impose any restrictions on this. For defects on the same material, it can be detected only once to obtain a defect image; it can also be detected multiple times by changing the scene data to obtain multiple defect images, which can be determined according to the needs, and the present embodiment does not impose any restrictions on this. There can be one or more defects in the same defect image. The image format (such as grayscale image, multi-spectral image, etc.) and file format (such as TIF, PNG, etc.) of the defect image can be determined according to the needs, and the present embodiment does not impose any restrictions on this.

[0077] The optical mode is a parameter information set for defect detection set on the hardware device before obtaining the defect image by using the hardware device to detect the material defect. The parameter information set may include one or more parameter information, and the parameter information may include but is not limited to the corresponding values ​​of the illumination light path, illumination light intensity, illumination polarization, collection light path, collection light intensity, collection polarization, or magnification. As long as there is one different parameter information in the parameter information set, it can be considered as a different type of optical mode.

[0078] The optical modes used to detect various types of materials may be the same or different, and may be determined according to needs, and this embodiment does not impose any limitation on this.

[0079] The scene data may be understood as information that has an impact on identifying defect information of a material, and may include, but is not limited to, one of optical modes, layer types, initial defect types, and the like.

[0080] In an example, each defect image may be used as a material defect image to be trained in the data to be trained; and one or more scene data corresponding to each defect image may be used as scene data in the data to be trained.

[0081] Assuming that each defect image corresponds to a layer type of scene data (for example, the metal layer of the wafer), each defect image can be used as a material defect image to be trained in the training data, and the metal layer of the wafer corresponding to each defect image can be used as the scene data, that is, each defect image corresponds to a type of scene data (the metal layer of the wafer).

[0082] Assuming that each defect image corresponds to a different optical mode, a different layer type, and a different initial defect type, each defect image can be used as a material defect image to be trained in the training data, and the optical mode, layer type, and initial defect type corresponding to each defect image can be used as scene data, that is, each defect image corresponds to three types of scene data.

[0083] For example, the three scene data corresponding to the material defect image A to be trained in the training data are: optical mode (illumination light path A1, illumination light intensity A2, illumination polarization A3, collection light path A4, collection light intensity A5, collection polarization A6, magnification A7), layer type (metal layer of the wafer), and initial defect type (particle contamination); the three scene data corresponding to the material defect image B to be trained in the training data are: optical mode (illumination light path A1, illumination light intensity A2, illumination polarization A3, collection light path B1, collection light intensity A5, collection polarization A6, magnification B2), layer type (oxide layer of the wafer), and initial defect type (crack).

[0084] Exemplarily, each material defect image to be trained in the training data may partially correspond to one type of scene data and partially correspond to multiple types of scene data. The scene data corresponding to each material defect image to be trained may be the same, different, or partially the same, and this embodiment does not impose any restrictions on this.

[0085] For example, the training data may be generated in the above manner and then input into the server. Alternatively, each defect image detected by the hardware device in the past may be used as the material defect image to be trained in the training data, and one or more scene data corresponding to each defect image may be used as the scene data in the training data, and input into the server. This embodiment does not limit the method for obtaining the training data, as long as the material defect image to be trained and at least one scene data in the training data can be obtained.

[0086] S102 , extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained.

[0087] Exemplarily, after acquiring the data to be trained, the server inputs the data to be trained into the initial model, and extracts features of each material defect image to be trained in the training data based on the initial model to obtain implicit features of each material defect image to be trained.

[0088] Among them, the implicit features represent the key information of the defects in the material defect image to be trained. The key information refers to a set of features that can effectively characterize and distinguish the defect types. For example, for a circular defect, its edge contour can be one of the key information because it can help determine the size and shape of the defect. For a crack defect, the interweaving mode of different branches and the particle distribution at the edge of the crack can be key information, which helps to identify the propagation path and severity of the crack.

[0089] S103. Extract features of the scene data according to the scene learner corresponding to the scene data in the initial model to obtain scene features of the scene data.

[0090] In one example, the initial model includes at least one scenario learner.

[0091] Exemplarily, the scene learner can be composed of a multi-layer perceptron (such as a feedforward neural network) or a self-attention mechanism network. The scene learner can learn scene features from scene data during the training of the initial model.

[0092] The scene feature indicates the relationship between the defect features in the material defect image to be trained and the scene data. It can also be understood that the scene feature reflects the change pattern of the defect features in the material defect image to be trained with the scene data.

[0093] It can be understood that, for the same defect, the corresponding defect features may be different under different scene data. For example, under a certain optical mode, the edge of the defect is clearer, so the edge features of the defect can be magnified in the scene features extracted under this optical mode; or the color of the defect is more obvious under a certain optical mode, so the color features of the defect can be magnified in the scene features extracted under this optical mode; for example, defects under certain layer types may appear as specific shapes, so in the scene features extracted under this layer type, the specific shape features of the defects can be magnified; for example, the defect type of particle contamination may appear as a circle or an ellipse, so in the scene features extracted under this defect type, the circular or elliptical features of the defect can be magnified.

[0094] In one example, the association between scene data and scene learners can be preset in the server, for example, optical mode G1 corresponds to scene learner J1, optical mode G2 corresponds to scene learner J2, layer type C1 corresponds to scene learner J3, layer type C2 corresponds to scene learner J4, initial defect type Q1 corresponds to scene learner J5, initial defect type Q2 corresponds to scene learner J6, and so on.

[0095] In a possible implementation, the material defect image to be trained corresponds to a type of scene data. Assuming that the material defect image D to be trained corresponds to a type of scene data is an optical mode G1, then:

[0096] For the optical mode G1 corresponding to the material defect image D to be trained, the optical mode G1 is input into the scene learner J1 corresponding to the optical mode G1 in the initial model for feature extraction to obtain the scene features of the optical mode G1.

[0097] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data. Assuming that the three scene data corresponding to the material defect image E to be trained are the optical mode G1, the layer type C2, and the initial defect type Q2, then:

[0098] For the optical mode G1 corresponding to the material defect image E to be trained, the optical mode G1 is input into the scene learner J1 corresponding to the optical mode G1 in the initial model for feature extraction to obtain the scene features of the optical mode G1;

[0099] For the layer type C2 corresponding to the material defect image E to be trained, the layer type C2 is input into the scene learner J4 corresponding to the layer type C2 in the initial model for feature extraction to obtain the scene feature of the layer type C2;

[0100] For the initial defect type Q2 corresponding to the material defect image E to be trained, the initial defect type Q2 is input into the scene learner J6 corresponding to the initial defect type Q2 in the initial model for feature extraction to obtain the scene features of the initial defect type Q2.

[0101] It is understandable that, for example, scene feature extraction can be performed for each scene data corresponding to each material defect image to be trained to obtain scene features of each scene data. The specific implementation method is similar to the implementation method of extracting scene features from the scene data corresponding to the material defect image D to be trained or the material defect image E to be trained, and will not be repeated here.

[0102] S104: Determine a recognition model based on the initial model, implicit features and scene features.

[0103] In one example, the data to be trained also includes calibrated defect information of each material defect image to be trained.

[0104] Exemplarily, for each material defect image to be trained in the training data, after obtaining the implicit features and at least one scene feature, the implicit features of each material defect image to be trained and the scene features of at least one scene data can be input into the initial model to obtain the predicted defect information of the material defect image to be trained. Then, the predicted defect information of the material defect image to be trained and the calibrated defect information are input into the preset loss function in the initial model to obtain the loss value. Then, back propagation is performed in the initial model according to the loss value to adjust the parameters in the initial model until the initial model reaches convergence, and the initial model is determined to be a recognition model.

[0105] The convergence of the initial model means that the initial model gradually stabilizes during the training process, the loss value no longer decreases significantly with the number of iterations, and the model performance (such as accuracy) no longer improves significantly. At this point, it can also be understood that the initial model has reached the optimal or near-optimal parameters.

[0106] The recognition model can be used, for example, to recognize defect information of the material to be recognized, wherein the defect information includes, for example, but is not limited to, one of defect equivalent size, defect type, etc. The meanings of defect equivalent size and defect type refer to the above introduction and will not be repeated here.

[0107] In one example, steps S101 to S104 may be repeated multiple times until the initial model reaches convergence, wherein different batches of data to be trained may be obtained each time step S101 is executed.

[0108] The model training method for defect information recognition provided by the embodiment of the present application obtains the data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data; the material defect image to be trained is a material defect image. Then, according to the initial model, feature extraction is performed on the material defect image to be trained to obtain implicit features of the material defect image to be trained. Then, according to the initial model, the scene data is input into the scene learner corresponding to the scene data in the initial model for feature extraction to obtain scene features of the scene data. Among them, the initial model includes at least one scene learner. Finally, the recognition model is determined based on the initial model, implicit features and scene features.

[0109] Since the initial model includes at least one scene learner, the scene data can be input into the scene learner corresponding to the scene data in the initial model for feature extraction according to the initial model, and the scene features of the scene data can be obtained, and then the initial model can be trained based on the scene features and implicit features. Therefore, it can simultaneously support model training of the data to be trained containing multiple scene data, with simple operation, fast training speed and high efficiency.

[0110] Figure 2 A flow chart of another model training method for defect information recognition provided in this application is as follows: Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, a model training method applied to defect information recognition is described in detail, and the method includes:

[0111] S201. Acquire data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data.

[0112] Exemplarily, the implementation of step S201 is similar to that of step S101. For a detailed description, please refer to the description of step S101, which will not be repeated here.

[0113] S202, based on the deep neural network unit in the initial model, feature extraction is performed on the material defect image to be trained to obtain implicit features of the material defect image to be trained; wherein the implicit features of the material defect image to be trained represent key information of defects in the material defect image to be trained

[0114] In one example, the initial model includes deep neural network units.

[0115] For example, it can be combined with Figure 3 To understand, Figure 3 The architecture diagram of a model for defect information identification provided in this application, assuming Figure 3 This is the architecture diagram of the initial model.

[0116] like Figure 3 As shown, Figure 3 The initial model 30 includes a deep neural network unit 301, a scene learner (such as but not limited to Figure 3 The scene learner 302, scene learner 304, scene learner 306 shown in the figure), each feature adjustment unit (such as but not limited to) connected to each scene learner Figure 3The feature adjustment unit 303, feature adjustment unit 305, feature adjustment unit 307 shown in the figure), the physical feature extraction unit 308, the fusion processing unit 309 connecting the physical feature extraction unit 308 and each feature adjustment unit, the task identification unit 310 and the task identification unit 311 connected to the fusion processing unit 309.

[0117] For example, after the server obtains the data to be trained, it inputs the data to be trained into Figure 3 In the initial model 30, based on Figure 3 The deep neural network unit in the initial model 30 performs feature extraction on each material defect image to be trained in the training data to obtain the implicit features of each material defect image to be trained. The meaning of the implicit features can be found in the description in step S102 and will not be repeated here. The deep neural network unit can be, for example, a convolutional neural network, including a visual geometry group network (Visual Geometry Group network, VGG), a residual network (Residual Network, ResNet), an inception network (Inception Network, Inception), etc., or a transformer model, including a vision transformer (Vision Transformer, ViT for short), etc.

[0118] S203. Perform feature extraction on the scene data according to the scene learner corresponding to the scene data in the initial model to obtain scene features of the scene data.

[0119] In one example, the initial model includes at least one scenario learner.

[0120] For example, the implementation of step S203 is similar to that of step S103. Based on the introduction of step S103, Figure 3 To understand.

[0121] In an example, it is assumed that the optical mode G1 corresponds to the scene learner J1 as Figure 3 The scene learner 302 in the layer type C2 corresponds to the scene learner J4: Figure 3 The scene learner 304 in the initial defect type Q2 corresponds to the scene learner J6: Figure 3 The scene learner 306 in .

[0122] In a possible implementation, the material defect image to be trained corresponds to a type of scene data. Assuming that the material defect image D to be trained corresponds to the optical mode G1, then:

[0123] For the optical mode G1 corresponding to the material defect image D to be trained, the optical mode G1 is input into Figure 3Feature extraction is performed in the scene learner 302 corresponding to the optical mode G1 in the initial model 30 to obtain the scene features of the material defect image D to be trained under the optical mode G1.

[0124] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data. Assuming that the three scene data corresponding to the material defect image E to be trained are the optical mode G1, the layer type C2, and the initial defect type Q2, then:

[0125] For the optical mode G1 corresponding to the material defect image E to be trained, the optical mode G1 is input into Figure 3 In the initial model 30, feature extraction is performed in the scene learner 302 corresponding to the optical mode G1 to obtain the scene features of the material defect image E to be trained under the optical mode G1;

[0126] For the layer type C2 corresponding to the material defect image E to be trained, the layer type C2 is input into Figure 3 In the initial model 30, feature extraction is performed in the scene learner 304 corresponding to the layer type C2 to obtain the scene features of the material defect image E to be trained under the layer type C2;

[0127] For the initial defect type Q2 corresponding to the material defect image E to be trained, the initial defect type Q2 is input into Figure 3 Feature extraction is performed in the scene learner 306 corresponding to the initial defect type Q2 in the initial model 30 to obtain the scene features of the material defect image E to be trained under the initial defect type Q2.

[0128] S204: Correct the implicit feature according to the scene feature to obtain a corrected implicit feature.

[0129] In one example, the scene feature represents the relationship between defects in the material defect image to be trained and the scene data.

[0130] For example, the meaning of the scene feature is described in step S103, which will not be repeated here. For example, the scene feature can be regarded as a set of weight parameters, and for each scene feature, the weight parameter and the implicit feature are corrected to obtain the corrected implicit feature.

[0131] The weight parameters are used to amplify the features in the implicit features that are consistent with the law of change of the defect features in the material defect image to be trained with the scene data, and to reduce the features in the implicit features that are inconsistent with the law of change of the defect features in the material defect image to be trained with the scene data. As a result, the corrected implicit features can be made more consistent with the law of change of the defect features in the material defect image to be trained with the scene data, and thus more accurate. Subsequent defect identification based on the more accurate corrected implicit features can also obtain more accurate defect information.

[0132] The correction processing of the weight parameter and the implicit feature can be achieved, for example, by multiplying the matrix of the weight parameter and the matrix of the implicit feature, or by adding the matrices, etc. This embodiment does not impose any restrictions on this, as long as the corrected implicit feature can achieve the above purpose.

[0133] For example, you can combine Figure 3 To understand, Figure 3 Each feature adjustment unit in the initial model 30 is used to correct the implicit features obtained by the deep neural network unit 301 according to the scene features obtained by the scene learner connected to the feature adjustment unit, so as to obtain the corrected implicit features.

[0134] Still taking the example in step S203 as the basis for the following example:

[0135] In a possible implementation, the material defect image to be trained corresponds to one kind of scene data. For example, in the example of step S203, the scene features of the material defect image D to be trained under the optical mode G1 are obtained by the scene learner 302 (assuming that the illumination light path of the ring illumination and the large-angle collection light path are included), and the material defect image D to be trained is subjected to feature extraction by the deep neural network unit 301. After obtaining the implicit features of the material defect image D to be trained, the scene features of the optical mode G1 and the implicit features of the material defect image D to be trained are input into the feature adjustment unit 303 to obtain the corrected implicit features of the material defect image D to be trained under the optical mode G1. Among them, the corrected implicit features take into account the influence of the scene features under the optical mode G1 on the implicit features, so that the defect information of the material defect image to be trained detected under the optical mode G1 can be more accurately identified based on the corrected implicit features. For example, when the scene features include the illumination light path of the ring illumination, the defect area usually appears as a bright bright spot, while the background is dark. Therefore, the corrected implicit features will enhance the features of these bright spots and improve the significance of edge and texture features. Furthermore, since scene features include larger angles of the light path, more scattered light can be captured, and therefore, the modified latent features enhance features that are sensitive to scattered light, such as sharp corners and turning points.

[0136] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data, and it is assumed that the deep neural network unit 301 has been used to extract features of the material defect image E to be trained, and the implicit features of the material defect image E to be trained have been obtained. For example, in the example of step S203, after the scene features of the material defect image E to be trained in the optical mode G1 are obtained by the scene learner 302, the scene features of the optical mode G1 and the implicit features of the material defect image E to be trained are input into the feature adjustment unit 303 to obtain the corrected implicit features of the material defect image E to be trained in the optical mode G1;

[0137] In the example of step S203, after the scene features of the material defect image E to be trained under the layer type C2 are obtained by the scene learner 304, the scene features of the layer type C2 and the implicit features of the material defect image E to be trained are input into the feature adjustment unit 305 to obtain the corrected implicit features of the material defect image E to be trained under the layer type C2;

[0138] In the example of step S203, after the scene features of the initial defect type Q2 corresponding to the material defect image E to be trained are obtained through the scene learner 306, the scene features of the initial defect type Q2 and the implicit features of the material defect image E to be trained are input into the feature adjustment unit 307 to obtain the corrected implicit features of the material defect image E to be trained under the initial defect type Q2.

[0139] S205, extracting image features from the material defect image to be trained based on an image processing method to obtain physical features of the material defect image to be trained, wherein the physical features of the material defect image to be trained represent at least one physical attribute of the defect in the material defect image to be trained.

[0140] Physical attributes can be understood as helping the initial model understand the nature and characteristics of defects. For example, grayscale value (the average grayscale value of each pixel in the material defect image to be trained) can indicate the brightness level of the defect area, and contrast can indicate the brightness difference between the defect and its surroundings.

[0141] In one example, step S205 includes the following steps:

[0142] The first step of step S205: extracting image features of the material defect image to be trained based on an image processing method to obtain initial image features of the material defect image to be trained;

[0143] The second step of step S205: if the training stage of the initial model is the pre-training stage, the initial image features of the material defect image to be trained are corrected according to a preset adjustment method of the physical features to obtain the physical features of the material defect image to be trained.

[0144] The third step of step S205: if the training stage of the initial model is the tuning stage, determining the initial image features of the material defect image to be trained as the physical features of the material defect image to be trained.

[0145] Exemplarily, the image processing method includes, but is not limited to, one or more of a Gray-Level Co-occurrence Matrix (GLCM) algorithm, calculating the average grayscale value of all pixels in the image, etc. For example, the Gray-Level Co-occurrence Matrix (GLCM) algorithm can be used to calculate the material defect image to be trained to obtain the initial image features of the material defect image to be trained (texture features of defects (such as contrast, uniformity, energy, etc.)), or the average grayscale value of all pixels in the material defect image to be trained is calculated to obtain the initial image features of the material defect image to be trained (brightness, etc.), or both image processing methods can be performed.

[0146] It is worth noting that the training phase of the initial model is divided into a pre-training phase and a tuning phase. The pre-training phase can be understood, for example, as inputting the data to be trained into the initial model, repeating steps S201-S207 for iterative training, until the initial model converges to obtain a pre-trained model. The tuning phase can be understood, for example, as inputting the data to be trained into the pre-trained model, repeating steps S201-S207 for iterative training, until the pre-trained model converges to obtain a recognition model. The convergence of the initial model has been introduced in step S104. For details, please refer to the description in step S104, which will not be repeated here.

[0147] The pre-trained model is used to extract common features and apply the common features to the recognition of various different defect information. The data to be trained used in the pre-training stage and the tuning stage are different. The data to be trained used in the tuning stage can be, for example, a data set related to the recognition of defect information in the material defect image to be identified. Then, the tuning stage adjusts the parameters of the pre-trained model through training, and the recognition model obtained can be more focused on accurately identifying the defect information of the material defect image to be identified.

[0148] A set of adjustment methods for initial preset physical features, the set including multiple adjustment methods for initial preset physical features, the adjustment methods for initial preset physical features can be understood as an image processing algorithm for adjusting the initial image features of the material defect image to be trained, for example, by setting a preset threshold to remove those features that are below or above a preset threshold in the initial image features of the material defect image to be trained, or by adding some features to the material defect image to be trained through image processing, or by creating new orthogonal features by linearly combining the initial image features of the material defect image to be trained.

[0149] The set of initial preset adjustment methods of physical characteristics can be pre-stored on a server, for example, and all the initial preset adjustment methods of physical characteristics in the set of initial preset adjustment methods of physical characteristics can be used as the preset adjustment methods of physical characteristics; it can also respond to a screening adjustment instruction and select a part or all of the initial preset adjustment methods of physical characteristics from the set of initial preset adjustment methods of physical characteristics as the preset adjustment methods of physical characteristics; wherein the screening adjustment instruction includes the initial preset adjustment methods of physical characteristics that need to be executed; it can also respond to a new adjustment instruction, wherein the new adjustment instruction includes the adjustment methods of physical characteristics, and use the adjustment methods of physical characteristics in the new adjustment instruction as the preset adjustment methods of physical characteristics; it can also respond to screening adjustment instructions, new adjustment instructions, and other methods at the same time.

[0150] The adjustment methods included in the preset physical feature adjustment methods can be selected based on experience, and this embodiment does not impose any restrictions on this, as long as it can highlight the features that are useful for identifying defect information and suppress the features that are useless for identifying defect information.

[0151] If it is determined that the training stage of the initial model is the pre-training stage, the initial image features of the material defect image to be trained are corrected according to a preset adjustment method of the physical features to obtain the physical features of the material defect image to be trained. The physical features represent at least one physical attribute of the defect in the material defect image to be trained.

[0152] If it is determined that the training stage of the initial model is the tuning stage, the initial image features of the material defect image to be trained are determined to be the physical features of the material defect image to be trained.

[0153] For example, you can combine Figure 3 To understand, Figure 3 The physical feature extraction unit 308 in the initial model 30 is used to execute the above steps in step S205.

[0154] Still taking the example in step S204 as the basis for the following example:

[0155] In a possible implementation, the material defect image to be trained corresponds to a type of scene data, and the material defect image D to be trained is input into the physical feature extraction unit 308 to obtain the physical features of the material defect image D to be trained.

[0156] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data, and the material defect image E to be trained is input into the physical feature extraction unit 308 to obtain the physical features of the material defect image E to be trained.

[0157] S206, fusing the physical feature with the corrected implicit feature to obtain a fused feature;

[0158] Exemplarily, for each material defect image to be trained, the physical features and the corrected implicit features are subjected to feature fusion processing to obtain fused features. The feature fusion processing can be performed by matrix addition or matrix multiplication of the physical features and the corrected implicit features, or feature fusion based on an attention mechanism, etc. This embodiment does not impose any restrictions on this, as long as the fused features can indicate the physical features and the corrected implicit features.

[0159] For example, you can combine Figure 3 To understand, Figure 3 The fusion processing unit 309 in the initial model 30 is used to execute the above steps in step S206.

[0160] Still taking the example in step S205 as the basis for the following example:

[0161] In a possible implementation, the material defect image to be trained corresponds to a type of scene data. After the corrected implicit features of the material defect image D to be trained in the optical mode G1 are obtained by the feature adjustment unit 303, and the physical features of the material defect image D to be trained are obtained by the physical feature extraction unit 308, the corrected implicit features of the material defect image D to be trained in the optical mode G1 and the physical features of the material defect image D to be trained are subjected to feature fusion processing to obtain the fused features of the material defect image D to be trained.

[0162] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data. After the physical features of the material defect image E to be trained are obtained by the physical feature extraction unit 308, and the corrected implicit features of the material defect image E to be trained under the optical mode G1 are obtained by the feature adjustment unit 303, the corrected implicit features of the material defect image E to be trained under the layer type C2 are obtained by the feature adjustment unit 305, and the corrected implicit features of the material defect image E to be trained under the initial defect type Q2 are obtained by the feature adjustment unit 307, the physical features of the material defect image E to be trained, the corrected implicit features of the material defect image E to be trained under the optical mode G1, the corrected implicit features of the material defect image E to be trained under the layer type C2, and the corrected implicit features of the material defect image E to be trained under the initial defect type Q2 are subjected to feature fusion processing to obtain the fused features of the material defect image E to be trained.

[0163] S207: Train the initial model according to the fusion features to obtain a recognition model.

[0164] In one example, the initial model includes multiple task identification units, and the data to be trained includes at least one calibrated defect information.

[0165] In one example, step S207 includes the following steps:

[0166] The first step of step S207 is: inputting the fused features into the task identification unit corresponding to the calibrated defect information in the initial model for processing to obtain the identified defect information.

[0167] The second step of step S207: obtaining the loss value of the task identification unit according to the calibrated defect information corresponding to the task identification unit and the identified defect information corresponding to the task identification unit.

[0168] The third step of step S207: If the training stage of the initial model is the pre-training stage, according to the loss value of the task identification unit, the parameters of each unit in the initial model except the task identification unit that does not correspond to the calibrated defect information and the scene learner that does not correspond to the scene data are adjusted to obtain a pre-trained model.

[0169] The fourth step of step S207: If the training stage of the initial model is the tuning stage, according to the loss value of the task identification unit, the parameters of each unit in the pre-trained model except the deep neural network unit, the task identification unit that does not correspond to the calibrated defect information, and the scene learner that does not correspond to the scene data are adjusted to obtain the recognition model.

[0170] Exemplarily, each defect in each material defect image to be trained corresponds to at least one calibrated defect information. The meaning of the defect information can be referred to the introduction in step S104, which will not be repeated here. The calibrated defect information can be understood as real defect information, for example, the material corresponding to the material defect image to be trained can be detected by a scanning electron microscope (Scanning Electron Microscope, referred to as SEM) or a confocal microscope (Confocal Microscopy), and the calibrated defect information corresponding to the defect in the material defect image to be trained can be obtained. This embodiment does not limit the method for obtaining the calibrated defect information, as long as the calibrated defect information can be obtained.

[0171] Each defect in each material defect image to be trained may correspond to, for example, one calibrated defect information (such as a calibrated defect equivalent size) or multiple calibrated defect information (such as a calibrated defect equivalent size and a calibrated defect type).

[0172] The initial model includes multiple task identification units, each of which is used to identify a type of defect information. For example, Figure 3 To understand, such as Figure 3 In the initial model 30, for example, the task identification unit 310 can be composed of a regression network for identifying the defect equivalent size. The task identification unit 311 can be composed of a classification network for identifying the defect type. The classification network can be, for example, an artificial neural network (such as a convolutional neural network, a residual network, etc.) for classification tasks. In various embodiments of the present application, the classification network can output different defect types, such as cracks, scratches, holes, etc.

[0173] The calibrated defect information may also include, for example, an identifier of a task identification unit, which indicates the task identification unit associated with the calibrated defect information. For example, the identifier of the calibrated defect equivalent size indicates that the task identification unit associated with the calibrated defect equivalent size is Figure 3 The task identification unit 310 in.

[0174] Still based on the example in step S206, the first step of step S207 is exemplified:

[0175] In a possible implementation, the material defect image to be trained corresponds to a scene data. Assume that the material defect image D to be trained includes a calibrated defect information, and the calibrated defect information indicates that the associated task identification unit is Figure 3 The task identification unit 310 in the figure can also be understood as the need to perform defect equivalent size identification training on the material defect image D to be trained.

[0176] In the example of step S206, after the fusion feature of the material defect image D to be trained is obtained by the fusion processing unit 309, the fusion feature of the material defect image D to be trained is input into Figure 3 The initial model 30 in the task identification unit 310 corresponding to the calibrated defect information is processed to obtain the defect equivalent size identified for the material defect image D to be trained.

[0177] In another possible implementation, the material defect image to be trained corresponds to a variety of scene data. Assume that the material defect image E to be trained includes two calibrated defect information (calibrated defect equivalent size E1 and calibrated defect type E2), wherein the calibrated defect equivalent size E1 indicates that the associated task identification unit is Figure 3 The task identification unit 310 in the example can also be understood as the need to train the defect equivalent size recognition of the material defect image E to be trained. The calibrated defect type E2 identifier indicates that the associated task identification unit is Figure 3 The task identification unit 311 in the training may also be understood as also needing to perform defect type identification training on the material defect image E to be trained.

[0178] In the example of step S206, after the fusion feature of the material defect image E to be trained is obtained by the fusion processing unit 309, the fusion feature of the material defect image E to be trained is input into Figure 3 The initial model 30 in the task identification unit 310 corresponding to the calibrated defect equivalent size E1 is processed to obtain the defect equivalent size identified by the material defect image E to be trained. At the same time, the fusion features of the material defect image E to be trained are input into Figure 3 The task identification unit 311 corresponding to the calibrated defect type E2 in the initial model 30 is processed to obtain the defect type identified for the material defect image E to be trained.

[0179] In one example, the fusion features of each material defect image to be trained in the training data can be input into the task identification unit corresponding to the calibrated defect information in the initial model for processing to obtain the identified defect information. The specific implementation method is similar to the implementation method of obtaining the identified defect information of the material defect image D to be trained and the material defect image E to be trained, and will not be repeated here.

[0180] Next, for example, in the second step of step S207, for example, for the task identification unit 310, each calibrated defect information associated with the task identification unit 310 in the training data, and the identified defect information identified by the task identification unit 310 and corresponding to each calibrated defect information, are input into the loss function in the task identification unit 310 to obtain the loss value of the task identification unit 310.

[0181] Then, for example, in the third step of step S207 , it is assumed that the scene data in the to-be-trained data corresponding to each calibrated defect information of the association task identification unit 310 is only associated with the scene learner 302 , the scene learner 304 , and the scene learner 306 .

[0182] If it is determined that the training phase of the initial model is the pre-training phase, then for the task identification unit 310, according to the loss value of the task identification unit 310, Figure 3 The deep neural network unit, physical feature extraction unit, scene learner 302, scene learner 304, scene learner 306, fusion processing unit 309, and task identification unit 310 in the initial model 30 are adjusted in parameters.

[0183] The implementation method of the loss value of the task identification unit 311 is similar to the implementation method of the loss value of the task identification unit 310 , and will not be repeated here.

[0184] It is assumed that the scene data in the to-be-trained data corresponding to each calibrated defect information of the associated task identification unit 311 is only associated with the scene learner 302 and the scene learner 304 .

[0185] For the task identification unit 311, according to the loss value of the task identification unit 311, Figure 3 The deep neural network unit, physical feature extraction unit, scene learner 302, scene learner 304, fusion processing unit 309, and task identification unit 311 in the initial model 30 are adjusted in parameters.

[0186] against Figure 3 All task identification units in the initial model 30 adjust the parameters of the initial model to obtain a pre-trained model. The implementation method of adjusting the parameters of the initial model is similar to the method of adjusting the parameters of the task identification unit 310 or the task identification unit 311, which will not be repeated here.

[0187] For example, in the fourth step of step S207, if it is determined that the training phase of the initial model is the tuning phase, the parameters of the pre-trained model are adjusted for all task recognition units in the pre-trained model to obtain a recognition model. The meaning of the recognition model is described in step S104 and will not be repeated here. When adjusting the parameters of the pre-trained model, the parameters of the deep neural network unit are frozen, and the other parameters are adjusted in the same way as in the pre-training phase.

[0188] The model training method for defect information recognition provided in the embodiment of the present application, since each scene data corresponds to a scene learner for extracting scene features, can avoid the mutual influence of different scene learners when adjusting parameters during multiple training iterations (for example, the parameters of scene learners F1 and F2 were adjusted in the last iteration, and the parameters of scene learners F2 and F3 were adjusted in this iteration. At this time, the parameters of scene learner F1 will not be modified, thereby ensuring the accuracy of the parameters of scene learner F1), so that the parameters of the scene learner in the recognition model are more accurate.

[0189] Based on the deep learning network, the material defect image to be trained is subjected to deep learning processing to obtain the implicit features of the material defect image to be trained, and then the implicit features are corrected according to the scene features to obtain the corrected implicit features. The corrected implicit features can be made more consistent with the changing rules of the defect features in the material defect image to be trained with the scene data, and thus more accurate; and based on the image processing method, image features are extracted from the material defect image to be trained to obtain the physical features of the material defect image to be trained, and then training is performed based on the more accurate corrected implicit features and physical features, so that a recognition model with more accurate parameters can be obtained.

[0190] Since the physical features are extracted from the material defect images to be trained based on image processing, the accuracy of the physical features is relatively high, which can accelerate the learning speed of the deep neural network units and each scene learner in the initial model, thereby improving the training efficiency of the initial model.

[0191] In the pre-training stage, the initial image features of the material defect image to be trained are corrected according to the preset physical feature adjustment method to obtain the physical features of the material defect image to be trained. The initial image features can be corrected based on experience to obtain more accurate physical features, and then training is performed based on the more accurate physical features to obtain a recognition model with more accurate parameters.

[0192] During the tuning stage, the image features of the material defect images to be trained are directly extracted based on the image processing method. The initial image features of the material defect images to be trained are used as physical features. The deep neural network units are frozen when the parameters are updated. This can speed up the training of the pre-trained model and improve the training efficiency.

[0193] Since the initial model includes multiple task identification units, it can simultaneously support the training of identifying one or more defect information.

[0194] Figure 4 A schematic diagram of a method for identifying defect information of a material provided in this application. Figure 1 Example or Figure 2 The recognition model obtained in the embodiment can be used for Figure 4 The embodiment performs identification of defect information, such as Figure 4 As shown, Figure 4 The method of the embodiment comprises:

[0195] S401, obtaining data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data.

[0196] Exemplarily, the implementation methods of step S410 and step S101 are similar, wherein the method for acquiring the material defect image to be identified in step S410 is similar to the method for acquiring the material defect image to be trained in step S101, and the method for acquiring the scene data in step S410 is similar to the method for acquiring the scene data in step S101. For details, please refer to the description in step S101, which will not be repeated here.

[0197] S402 , extracting features of the material defect image to be identified according to the identification model to obtain implicit features of the material defect image to be identified.

[0198] Exemplarily, after acquiring the data to be identified, the server inputs the data to be identified into the recognition model, and extracts features of each material defect image to be identified in the data to be identified based on the recognition model to obtain implicit features of each material defect image to be identified.

[0199] The implicit features represent key information of defects in the material defect image to be identified. The meaning of the key information has been introduced in step S102. For details, please refer to the description in step S102, which will not be repeated here.

[0200] The recognition model can be based on Figure 1 Example or Figure 2 The training method of the embodiment obtains the recognition model.

[0201] S403: Extract features of the scene data according to the scene learner corresponding to the scene data in the recognition model to obtain scene features of the scene data.

[0202] In one example, the recognition model includes at least one scene learner.

[0203] Among them, the meanings of scene learner and scene features are similar to those of scene learner and scene features in step S103. For details, please refer to the description in step S103, which will not be repeated here.

[0204] In one example, the association between scene data and scene learners can be preset in the server, for example, optical mode G1 corresponds to scene learner J1, optical mode G2 corresponds to scene learner J2, layer type C1 corresponds to scene learner J3, layer type C2 corresponds to scene learner J4, initial defect type Q1 corresponds to scene learner J5, initial defect type Q2 corresponds to scene learner J6, and so on.

[0205] In a possible implementation, the material defect image to be identified corresponds to one type of scene data. Assuming that the one type of scene data corresponding to the material defect image K to be identified is the optical mode G1, then:

[0206] For the optical mode G1 corresponding to the material defect image K to be identified, the optical mode G1 is input into the scene learner J1 corresponding to the optical mode G1 in the recognition model for feature extraction to obtain the scene features of the material defect image K to be identified under the optical mode G1.

[0207] In another possible implementation, the material defect image to be identified corresponds to a plurality of scene data. Assuming that the three scene data corresponding to the material defect image L to be identified are the optical mode G1, the layer type C2, and the initial defect type Q2, then:

[0208] For the optical mode G1 corresponding to the material defect image L to be identified, the optical mode G1 is input into the scene learner J1 corresponding to the optical mode G1 in the recognition model for feature extraction, so as to obtain the scene features of the material defect image L to be identified under the optical mode G1;

[0209] For the layer type C2 corresponding to the material defect image L to be identified, the layer type C2 is input into the scene learner J4 corresponding to the layer type C2 in the recognition model for feature extraction, so as to obtain the scene features of the material defect image L to be identified under the layer type C2;

[0210] For the initial defect type Q2 corresponding to the material defect image L to be identified, the initial defect type Q2 is input into the scene learner J6 corresponding to the initial defect type Q2 in the recognition model for feature extraction to obtain the scene features of the material defect image L to be identified under the initial defect type Q2.

[0211] It is understandable that, for example, scene feature extraction can be performed for each scene data corresponding to each material defect image to be identified to obtain the scene feature of each scene data. The specific implementation method is similar to the implementation method of extracting scene features from the scene data corresponding to the material defect image K to be identified or the material defect image L to be identified, and will not be repeated here.

[0212] S404: Determine defect information of the material defect image to be identified based on the recognition model, scene features, and implicit features.

[0213] Exemplarily, for each material defect image to be identified in the data to be identified, after obtaining the implicit feature and at least one scene feature, the implicit feature and the at least one scene feature can be processed based on the recognition model to obtain defect information of the material defect image to be identified.

[0214] Still taking the example in step S403 as an example, the following is described:

[0215] In one possible implementation, the material defect image to be identified corresponds to a type of scene data. It is assumed that the material defect image K to be identified has obtained the implicit features of the material defect image K to be identified, and the scene features of the material defect image K to be identified under the optical mode G1 have been obtained in step S403. Then, based on the recognition model, the implicit features of the material defect image K to be identified and the scene features of the material defect image K to be identified under the optical mode G1 are processed to obtain the defect information of the material defect image L to be identified.

[0216] In another possible implementation, the material defect image to be identified corresponds to multiple scene data. It is assumed that the material defect image L to be identified has obtained the implicit features of the material defect image L to be identified, and in step S403, the scene features of the material defect image L to be identified under the optical mode G1, the scene features of the material defect image L to be identified under the layer type C2, and the scene features of the material defect image L to be identified under the initial defect type Q2 have been obtained. Then, based on the recognition model, the implicit features of the material defect image L to be identified, the scene features of the material defect image L to be identified under the optical mode G1, the scene features of the material defect image L to be identified under the layer type C2, and the scene features of the material defect image L to be identified under the initial defect type Q2 are processed to obtain the defect information of the material defect image L to be identified.

[0217] The defect information includes, for example, but is not limited to, one of a defect equivalent size, a defect type, etc. The defect equivalent size may be, for example, the diameter of a cylindrical defect having the same scattered light intensity as the defect in the material. The meaning of the defect type is described in step S401 and will not be repeated here.

[0218] The material defect information identification method provided in the embodiment of the present application obtains data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data; the material defect image to be identified is a material defect image. Then, feature extraction is performed on the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified. Among them, the recognition model includes at least one scene learner. Afterwards, feature extraction is performed on the scene data according to the scene learner corresponding to the scene data in the recognition model to obtain scene features of the scene data. Furthermore, based on the recognition model, the scene features and the implicit features, the defect information of the material defect image to be identified is determined.

[0219] Since the recognition model includes at least one scene learner, the scene data can be input into the scene learner corresponding to the scene data in the recognition model for feature extraction according to the recognition model to obtain the scene features of the scene data, and then the scene features and implicit features are processed to obtain the defect information of the material defect image to be identified. Therefore, it can simultaneously support accurate defect identification of the data to be identified containing multiple scene data, and the operation is simple, the recognition speed is fast, and the efficiency is high.

[0220] Figure 5 A flow chart of another material defect information identification method provided in this application, such as Figure 5 As shown, in this embodiment Figure 4 Based on the embodiment, a method for identifying defect information of a material is described in detail, and the method includes:

[0221] S501. Acquire data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one type of scene data.

[0222] Exemplarily, the implementation of step S501 is similar to that of step S401 , and for detailed description, please refer to the description of step S401 , which will not be repeated here.

[0223] S502: Based on the deep neural network unit in the recognition model, feature extraction is performed on the material defect image to be identified to obtain implicit features of the material defect image to be identified, wherein the implicit features represent key information of defects in the material defect image to be identified.

[0224] In one example, the recognition model includes a deep neural network unit.

[0225] For example, it can be combined with Figure 3 To understand, assume Figure 3 This is the architecture diagram of the recognition model.

[0226] like Figure 3 As shown, Figure 3 The recognition model 30 includes a deep neural network unit 301, a plurality of scene learners (such as but not limited to Figure 3 The scene learner 302, scene learner 304, scene learner 306 shown in the figure), each feature adjustment unit (such as but not limited to) connected to each scene learner Figure 3 The feature adjustment unit 303, feature adjustment unit 305, feature adjustment unit 307 shown in the figure), the physical feature extraction unit 308, the fusion processing unit 309 connecting the physical feature extraction unit 308 and each feature adjustment unit, the task identification unit 310 and the task identification unit 311 connected to the fusion processing unit 309.

[0227] For example, after the server obtains the data to be identified, it inputs the data to be identified into Figure 3 In the recognition model 30, based on Figure 3 The deep neural network unit in the recognition model 30 extracts features from each material defect image to be identified in the identification data to obtain implicit features of each material defect image to be identified. The meaning of the implicit features can be found in the description of step S402, which will not be repeated here. The introduction of the deep neural network unit can be found in the description of step S202, which will not be repeated here.

[0228] S503: Extract features of the scene data according to the scene learner corresponding to the scene data in the recognition model to obtain scene features of the scene data.

[0229] In one example, the recognition model includes at least one scene learner.

[0230] For example, the implementation of step S503 is similar to that of step S403. Based on the introduction of step S403, Figure 3 To understand.

[0231] In an example, it is assumed that the optical mode G1 corresponds to the scene learner J1 as Figure 3 The scene learner 302 in the layer type C2 corresponds to the scene learner J4: Figure 3 The scene learner 304 in the initial defect type Q2 corresponds to the scene learner J6: Figure 3 The scene learner 306 in .

[0232] In a possible implementation, the material defect image to be identified corresponds to one type of scene data. Assuming that the one type of scene data corresponding to the material defect image K to be identified is the optical mode G1, then:

[0233] For the optical mode G1 corresponding to the material defect image K to be identified, the optical mode G1 is input into Figure 3 Feature extraction is performed in the scene learner 302 corresponding to the optical mode G1 in the recognition model 30 to obtain the scene features of the material defect image K to be identified under the optical mode G1.

[0234] In another possible implementation, the material defect image to be identified corresponds to a plurality of scene data. Assuming that the three scene data corresponding to the material defect image L to be identified are the optical mode G1, the layer type C2, and the initial defect type Q2, then:

[0235] For the optical mode G1 corresponding to the material defect image L to be identified, the optical mode G1 is input into Figure 3 In the recognition model 30, feature extraction is performed in the scene learner 302 corresponding to the optical mode G1 to obtain scene features of the material defect image L to be identified under the optical mode G1;

[0236] For the layer type C2 corresponding to the material defect image L to be identified, the layer type C2 is input into Figure 3 In the recognition model 30, feature extraction is performed in the scene learner 304 corresponding to the layer type C2 to obtain the scene features of the material defect image L to be identified under the layer type C2;

[0237] For the initial defect type Q2 corresponding to the material defect image L to be identified, the initial defect type Q2 is input into Figure 3 Feature extraction is performed in the scene learner 306 corresponding to the initial defect type Q2 in the recognition model 30 to obtain the scene features of the material defect image L to be identified under the initial defect type Q2.

[0238] S504: Correct the implicit feature according to the scene feature to obtain a corrected implicit feature.

[0239] In one example, the scene feature is used to characterize the relationship between the defects in the material defect image to be identified and the scene data.

[0240] For example, the meaning of the scene feature is described in step S403, which will not be repeated here. For example, the scene feature can be regarded as a set of weight parameters, and the modified implicit feature can be obtained by modifying the weight parameters and the implicit feature.

[0241] The weight parameters are used to amplify the features in the implicit features that are consistent with the law of change of the defect features in the material defect image to be identified with the scene data, and to reduce the features in the implicit features that are inconsistent with the law of change of the defect features in the material defect image to be identified with the scene data. As a result, the corrected implicit features can be made more consistent with the law of change of the defect features in the material defect image to be identified with the scene data, and thus more accurate. Subsequent defect identification based on the more accurate corrected implicit features can also obtain more accurate defect information.

[0242] The correction processing of the weight parameter and the implicit feature can be achieved, for example, by multiplying the matrix of the weight parameter and the matrix of the implicit feature, or by adding the matrices, etc. This embodiment does not impose any restrictions on this, as long as the corrected implicit feature can achieve the above purpose.

[0243] For example, you can combine Figure 3 To understand, Figure 3 Each feature adjustment unit in the recognition model 30 is used to correct the implicit features obtained by the deep neural network unit 301 according to the scene features obtained by the scene learner connected to the feature adjustment unit, so as to obtain the corrected implicit features.

[0244] Still taking the example in step S503 as the basis for the following example:

[0245] In one possible implementation, the material defect image to be identified corresponds to one type of scene data. For example, in the example of step S503, the scene features of the material defect image K to be identified under the optical mode G1 are obtained by the scene learner 302, and the deep neural network unit 301 performs feature extraction on the material defect image K to be identified. After the implicit features of the material defect image K to be identified are obtained, the scene features of the optical mode G1 and the implicit features of the image K to be identified are input into the feature adjustment unit 303 to obtain the corrected implicit features of the material defect image K to be identified under the optical mode G1.

[0246] In another possible implementation, the material defect image to be identified corresponds to a variety of scene data, and it is assumed that the deep neural network unit 301 has been used to extract features of the material defect image E to be identified, and the implicit features of the image L to be identified are obtained. For example, in the example of step S503, after the scene features of the material defect image L to be identified in the optical mode G1 are obtained by the scene learner 302, the scene features of the optical mode G1 and the implicit features of the image E to be identified are input into the feature adjustment unit 303 to obtain the corrected implicit features of the material defect image L to be identified in the optical mode G1;

[0247] In the example of step S503, after the scene features of the material defect image L to be identified under the layer type C2 are obtained by the scene learner 304, the scene features of the layer type C2 and the implicit features of the material defect image L to be identified are input into the feature adjustment unit 305 to obtain the corrected implicit features of the material defect image L to be identified under the layer type C2;

[0248] In the example of step S503, after the scene features of the material defect image L to be identified under the initial defect type Q2 are obtained by the scene learner 306, the scene features of the initial defect type Q2 and the implicit features of the material defect image L to be identified are input into the feature adjustment unit 307 to obtain the corrected implicit features of the material defect image L to be identified under the initial defect type Q2.

[0249] S505: Extract image features of the material defect image to be identified based on an image processing method to obtain physical features of the material defect image to be identified. The physical features of the material defect image to be identified represent important attributes of the defects in the material defect image to be identified.

[0250] Exemplarily, the introduction of the image processing method is detailed in the description of step S205, which will not be repeated here. For example, the material defect image to be identified can be calculated by the Gray-Level Co-occurrence Matrix (GLCM) algorithm to obtain the physical characteristics of the material defect image to be identified (texture characteristics of defects (such as contrast, uniformity, energy, etc.)), or the material defect image to be identified is calculated by the average gray value of all pixels in the image to obtain the physical characteristics of the material defect image to be identified (brightness, etc.), or both image processing methods can be performed.

[0251] For example, you can combine Figure 3 To understand, Figure 3 The physical feature extraction unit 308 in the recognition model 30 is used to execute the above steps in step S505.

[0252] Still taking the example in step S504 as the basis for the following example:

[0253] In a possible implementation, the material defect image to be identified corresponds to one type of scene data, and the material defect image K to be identified is input into the physical feature extraction unit 308 to obtain the physical features of the material defect image K to be identified.

[0254] In another possible implementation, the material defect image to be identified corresponds to a variety of scene data, and the material defect image L to be identified is input into the physical feature extraction unit 308 to obtain the physical features of the material defect image L to be identified.

[0255] S506: Fusing the physical feature with the corrected implicit feature to obtain a fused feature.

[0256] Exemplarily, for each material defect image to be identified, the physical feature and each corrected implicit feature are subjected to feature fusion processing to obtain a fused feature. The feature fusion processing can be performed by matrix addition or matrix multiplication of the physical feature and each corrected implicit feature, or feature fusion based on an attention mechanism, etc. This embodiment does not impose any restrictions on this, as long as the fused feature can indicate the physical feature and each corrected implicit feature.

[0257] For example, you can combine Figure 3 To understand, Figure 3 The fusion processing unit 309 in the recognition model 30 is used to execute the above steps in step S506.

[0258] Still taking the example in step S505 as the basis for the following example:

[0259] In one possible implementation, the material defect image to be identified corresponds to one type of scene data. After the corrected implicit features of the material defect image K to be identified in the optical mode G1 are obtained by the feature adjustment unit 303, and the physical features of the material defect image K to be identified are obtained by the physical feature extraction unit 308, the corrected implicit features of the material defect image K to be identified in the optical mode G1 and the physical features of the material defect image K to be identified are subjected to feature fusion processing to obtain the fused features of the material defect image K to be identified.

[0260] In another possible implementation, the material defect image to be identified corresponds to multiple scene data. After the physical features of the material defect image L to be identified are obtained by the physical feature extraction unit 308, and the corrected implicit features of the material defect image L to be identified under the optical mode G1 are obtained by the feature adjustment unit 303, the corrected implicit features of the material defect image L to be identified under the layer type C2 are obtained by the feature adjustment unit 305, and the corrected implicit features of the material defect image L to be identified under the initial defect type Q2 are obtained by the feature adjustment unit 307, the physical features of the material defect image L to be identified, the corrected implicit features of the material defect image L to be identified under the optical mode G1, the corrected implicit features of the material defect image L to be identified under the layer type C2, and the corrected implicit features of the material defect image L to be identified under the initial defect type Q2 are subjected to feature fusion processing to obtain the fused features of the material defect image L to be identified.

[0261] S507: Input the fused features into the task identification unit corresponding to the identifier of the defect information to be identified in the identification model for processing, so as to obtain the defect information of the material defect image to be identified.

[0262] In one example, the recognition model includes a plurality of task recognition units, and the data to be recognized indicates the identifier of the defect information to be recognized.

[0263] Exemplarily, each task identification unit is used to identify a type of defect information, for example, Figure 3 To understand, such as Figure 3 In the recognition model 30, for example, the task recognition unit 310 may be composed of a regression network for recognizing the defect equivalent size. The task recognition unit 311 may be composed of a classification network for recognizing the defect type.

[0264] The identification of the defect information to be identified indicates the task identification unit associated with the defect information to be identified. For example, the identification of the defect equivalent size to be identified indicates that the task identification unit associated with the defect equivalent size to be identified is Figure 3 The task identification unit 310 in.

[0265] The data to be identified corresponding to each material defect image to be identified may include one or more identifiers of defect information to be identified.

[0266] Still taking the example in step S506 as the basis for the following example:

[0267] In a possible implementation, the material defect image to be identified corresponds to one type of scene data. Assume that the identification of the defect information to be identified in the data to be identified corresponding to the material defect image K to be identified indicates that the associated task identification unit is Figure 3 The task identification unit 310 in the embodiment can also be understood as the need to perform defect equivalent size identification on the material defect image K to be identified.

[0268] In the example of step S506, after the fusion feature of the material defect image K to be identified is obtained by the fusion processing unit 309, the fusion feature of the material defect image K to be identified is input into Figure 3 The task identification unit 310 corresponding to the identification of the defect information to be identified in the identification model 30 is processed to obtain the defect equivalent size identified for the material defect image K to be identified.

[0269] In another possible implementation, the material defect image to be identified corresponds to a variety of scene data. Assume that the data to be identified corresponding to the material defect image L to be identified includes two identifiers of defect information to be identified (defect equivalent size identifier L1 and defect type identifier L2), wherein the defect equivalent size identifier L1 indicates that the associated task identification unit is Figure 3 The task identification unit 310 in the example can also be understood as requiring the identification of the defect equivalent size of the material defect image L to be identified, and the defect type identifier L2 indicates that the associated task identification unit is Figure 3 The task identification unit 311 in the image may also be understood to also need to perform defect type identification on the material defect image L to be identified.

[0270] In the example of step S506, after the fusion feature of the material defect image L to be identified is obtained by the fusion processing unit 309, the fusion feature of the material defect image L to be identified is input into Figure 3 The task identification unit 310 corresponding to the defect equivalent size mark L1 in the recognition model 30 in the embodiment is processed to obtain the defect equivalent size identified by the material defect image L to be identified. At the same time, the fusion features of the material defect image L to be identified are input into Figure 3 The task identification unit 311 corresponding to the defect type identifier L2 in the recognition model 30 is processed to obtain the defect type identified for the material defect image L to be identified.

[0271] In one example, the fusion features of each material defect image to be identified in the data to be identified can be input into the task identification unit corresponding to the identifier of the defect information to be identified in the recognition model for processing to obtain the identified defect information. The specific implementation method is similar to the implementation method of obtaining the identified defect information from the material defect image K to be identified and the material defect image L to be identified, and will not be repeated here.

[0272] The material defect information identification method provided in the embodiment of the present application is based on the fusion processing of the corrected implicit features (which conform to the law of change of the defect features in the material defect image to be identified with the scene data) and the physical features, and thus, more fusion features are obtained. Therefore, defect identification based on the fusion features with more features can identify more accurate defect information. Moreover, the fusion features are not limited to extracting only grayscale values, and thus, the problem of inaccurate defect identification when the full well phenomenon occurs can be avoided by only extracting grayscale values ​​in the material defect image to be identified for defect identification.

[0273] Since the recognition model includes multiple task recognition units, it can simultaneously support the recognition of one or more defect information.

[0274] Figure 6 A schematic diagram of the structure of the defect information identification device provided in this application, such as Figure 6 As shown, the material defect information identification device 60 provided in this embodiment includes:

[0275] An acquisition module 601 is used to acquire data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data;

[0276] The extraction module 602 is used to extract features of the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified; wherein the recognition model includes at least one scene learner; and according to the scene learner corresponding to the scene data in the recognition model, extract features of the scene data to obtain scene features of the scene data;

[0277] The recognition module 603 is used to recognize the implicit features and scene features based on the recognition model to obtain the defect information of the material defect image to be recognized.

[0278] In a possible implementation, the scene feature is used to characterize the relationship between the defect in the material defect image to be identified and the scene data;

[0279] In a possible implementation, the identification module 603 is specifically configured to:

[0280] Correcting the latent features according to the scene features to obtain corrected latent features;

[0281] The corrected implicit features are identified based on the recognition model to obtain defect information of the material defect image to be identified.

[0282] In a possible implementation manner, the “recognizing the corrected implicit features based on the recognition model to obtain defect information of the material defect image to be recognized” in the recognition module 603 is specifically used to:

[0283] Extracting image features of the material defect image to be identified based on an image processing method to obtain physical features of the material defect image to be identified; wherein the physical features of the material defect image to be identified represent important attributes of the defects in the material defect image to be identified;

[0284] The physical features are fused with the corrected implicit features to obtain fused features; and the fused features are recognized based on the recognition model to obtain defect information of the material defect image to be recognized.

[0285] In a possible implementation, the recognition model includes a plurality of task recognition units, and the data to be recognized indicates the identification of the defect information to be recognized;

[0286] In a possible implementation, “identifying the fused features based on the recognition model to obtain defect information of the material defect image to be identified” in the recognition module 603 includes:

[0287] The fused features are input into the task identification unit corresponding to the identification of the defect information to be identified in the identification model for processing, so as to obtain the defect information of the material defect image to be identified.

[0288] In one possible implementation, the recognition model includes a deep neural network unit;

[0289] In a possible implementation, the “extracting features of the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified” in the extraction module 602 is specifically used to:

[0290] Based on the deep neural network unit in the recognition model, feature extraction is performed on the material defect image to be identified to obtain implicit features of the material defect image to be identified; wherein the implicit features of the material defect image to be identified represent key information of the defects in the material defect image to be identified. The material defect information identification device provided in this embodiment can execute the method provided in the above-mentioned material defect information identification method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0291] Figure 7 A schematic diagram of the structure of a model training device for defect information recognition provided in this application, such as Figure 7 As shown, the model training device 70 for defect information recognition provided in this embodiment includes:

[0292] An acquisition module 701 is used to acquire data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data;

[0293] The extraction module 702 is used to extract features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained; wherein the initial model includes at least one scene learner; and according to the scene learner corresponding to the scene data in the initial model, extract features of the scene data to obtain scene features of the scene data;

[0294] The training module 703 is used to train the initial model according to implicit features and scene features to obtain a recognition model; wherein the recognition model is used to identify the material defect images to be identified in the first aspect of the invention content and / or various possible implementation methods of the first aspect to obtain defect information.

[0295] In a possible implementation, the scene feature represents the relationship between the defect in the material defect image to be trained and the scene data;

[0296] In a possible implementation, the training module 703 is specifically configured to:

[0297] Correcting the latent features according to the scene features to obtain corrected latent features;

[0298] According to the modified implicit features, the initial model is trained to obtain the recognition model.

[0299] In a possible implementation, the “training the initial model according to the modified implicit features to obtain the recognition model” in the training module 703 is specifically used to:

[0300] Extracting image features of the material defect image to be trained based on an image processing method to obtain physical features of the material defect image to be trained; wherein the physical features of the material defect image to be trained represent important attributes of defects in the material defect image to be trained;

[0301] The physical features are fused with the modified implicit features to obtain fused features; and the initial model is trained based on the fused features to obtain a recognition model.

[0302] In a possible implementation manner, the “extracting image features from the material defect image to be trained based on an image processing method to obtain physical features of the material defect image to be trained” in the training module 703 is specifically used to:

[0303] Extracting image features of the material defect image to be trained based on an image processing method to obtain initial image features of the material defect image to be trained;

[0304] If the training stage of the initial model is a pre-training stage, the initial image features of the material defect image to be trained are corrected according to a preset adjustment method of the physical features to obtain the physical features of the material defect image to be trained;

[0305] If the training phase of the initial model is a tuning phase, the initial image features of the material defect image to be trained are determined to be the physical features of the material defect image to be trained.

[0306] In a possible implementation, the initial model includes a plurality of task identification units, and the data to be trained includes at least one calibrated defect information;

[0307] In a possible implementation manner, “training the initial model according to the fusion features to obtain the recognition model” in the training module 703 is specifically used to:

[0308] The fused features are input into the task identification unit corresponding to the calibrated defect information in the initial model for processing to obtain the identified defect information;

[0309] Obtaining a loss value of the task identification unit according to the calibrated defect information corresponding to the task identification unit and the identified defect information corresponding to the task identification unit;

[0310] According to the loss value of the task recognition unit, the parameters of the initial model are adjusted to obtain the recognition model.

[0311] In a possible implementation, “adjusting parameters of the initial model according to the loss value of the task identification unit to obtain the identification model” in the training module 703 is specifically used to:

[0312] If the training stage of the initial model is the pre-training stage, according to the loss value of the task recognition unit, the parameters of each unit in the initial model except the task recognition unit that does not correspond to the calibrated defect information and the scene learner that does not correspond to the scene data are adjusted to obtain a pre-training model;

[0313] If the training stage of the initial model is the tuning stage, according to the loss value of the task recognition unit, the parameters of each unit in the pre-trained model except the deep neural network unit, the task recognition unit that does not correspond to the calibrated defect information, and the scene learner that does not correspond to the scene data are adjusted to obtain the recognition model.

[0314] In one possible implementation, the initial model includes a deep neural network unit;

[0315] In a possible implementation manner, the “extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained” in the extraction module 702 is specifically used to:

[0316] Based on the deep neural network unit in the initial model, feature extraction is performed on the material defect image to be trained to obtain implicit features of the material defect image to be trained; wherein the implicit features of the material defect image to be trained represent key information of defects in the material defect image to be trained.

[0317] The model training device for defect information recognition provided in this embodiment can execute the method provided in the above-mentioned model training method embodiment for defect information recognition. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0318] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0319] In the specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method. The specific implementation process of the processor 801 can refer to the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0320] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0321] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0322] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0323] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0324] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0325] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0326] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0327] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0328] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0329] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0330] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0331] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for identifying defect information of a material, characterized in that: include: Acquire data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data; Extracting features of the material defect image to be identified according to the recognition model to obtain implicit features of the material defect image to be identified; wherein the recognition model includes at least one scene learner; Extracting features of the scene data according to a scene learner corresponding to the scene data in the recognition model to obtain scene features of the scene data; Defect information of the material defect image to be identified is determined based on the recognition model, the scene features, and the implicit features.

2. The method according to claim 1, characterized in that The scene feature is used to characterize the relationship between the defect in the material defect image to be identified and the scene data; Determining defect information of the material defect image to be identified based on the recognition model, the scene feature, and the implicit feature includes: The material defect image to be identified performs correction processing on the implicit feature according to the scene feature to obtain a corrected implicit feature; Based on the recognition model and the corrected implicit features, defect information of the material defect image to be recognized is determined.

3. The method according to claim 2, characterized in that The step of determining defect information of the material defect image to be identified based on the recognition model and the corrected implicit features includes: Performing image feature extraction on the material defect image to be identified based on an image processing method to obtain physical features of the material defect image to be identified; wherein the physical features of the material defect image to be identified represent at least one physical attribute of the defect in the material defect image to be identified; The physical feature is fused with the corrected implicit feature to obtain a fused feature; and the fused feature is identified based on the identification model to obtain defect information of the material defect image to be identified.

4. The method according to claim 3, characterized in that The identification model includes a plurality of task identification units, and the data to be identified indicates the identification of the defect information to be identified; The fusion feature is identified based on the identification model to obtain defect information of the material defect image to be identified, including: The fusion feature is input into the task identification unit corresponding to the identifier of the defect information to be identified in the identification model for processing, so as to obtain the defect information of the material defect image to be identified.

5. The method according to any one of claims 1 to 4, characterized in that The recognition model includes a deep neural network unit; The extracting features of the material defect image to be identified according to the identification model to obtain implicit features of the material defect image to be identified includes: Based on the deep neural network unit in the recognition model, feature extraction is performed on the material defect image to be identified to obtain implicit features of the material defect image to be identified; wherein the implicit features of the material defect image to be identified represent key information of the defects in the material defect image to be identified.

6. A model training method for defect information recognition, characterized in that: include: Acquire data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data; Extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained; wherein the initial model includes at least one scene learner; and extracting features of the scene data according to the scene learner corresponding to the scene data in the initial model to obtain scene features of the scene data; A recognition model is determined according to the initial model, the implicit features and the scene features; wherein the recognition model is used to identify the material defect image to be identified to obtain defect information.

7. The method according to claim 6, characterized in that The scene feature represents the relationship between the defects in the material defect image to be trained and the scene data; The determining of the recognition model according to the initial model, the implicit features and the scene features includes: Correcting the latent feature according to the scene feature to obtain a corrected latent feature; The recognition model is determined according to the initial model and the modified implicit features.

8. The method according to claim 7, characterized in that Determining the recognition model according to the initial model and the modified implicit features includes: Based on the image processing method, image features are extracted from the material defect image to be trained to obtain physical features of the material defect image to be trained; wherein the physical features of the material defect image to be trained represent the defects in the material defect image to be trained; the physical features are fused with the corrected implicit features to obtain fused features; and according to the fused features, the initial model is trained to obtain a recognition model.

9. The method according to claim 8, characterized in that The extracting image features of the material defect image to be trained based on the image processing method to obtain the physical features of the material defect image to be trained includes: Extracting image features of the material defect image to be trained based on an image processing method to obtain initial image features of the material defect image to be trained; If the training stage of the initial model is a pre-training stage, the initial image features of the material defect image to be trained are corrected according to a preset adjustment method of the physical features to obtain the physical features of the material defect image to be trained; If the training phase of the initial model is a tuning phase, the initial image features of the material defect image to be trained are determined to be physical features of the material defect image to be trained.

10. The method according to claim 8, characterized in that The initial model includes a plurality of task identification units, and the data to be trained includes at least one calibrated defect information; The step of training the initial model according to the fusion features to obtain a recognition model includes: Inputting the fused features into the task identification unit corresponding to the calibrated defect information in the initial model for processing to obtain identified defect information; Obtaining a loss value of the task identification unit according to the calibrated defect information corresponding to the task identification unit and the identified defect information corresponding to the task identification unit; According to the loss value of the task identification unit, the parameters of the initial model are adjusted to obtain a recognition model.

11. The method according to claim 10, characterized in that The step of adjusting the parameters of the initial model according to the loss value of the task identification unit to obtain the identification model includes: If the training stage of the initial model is a pre-training stage, according to the loss value of the task identification unit, parameters of each unit in the initial model except for the task identification unit that does not correspond to the calibrated defect information and the scene learner that does not correspond to the scene data are adjusted to obtain a pre-training model; If the training stage of the initial model is a tuning stage, according to the loss value of the task identification unit, the parameters of each unit in the pre-trained model except the deep neural network unit, the task identification unit that does not correspond to the calibrated defect information, and the scene learner that does not correspond to the scene data are adjusted to obtain a recognition model.

12. The method according to any one of claims 6 to 11, characterized in that: The initial model includes a deep neural network unit; The extracting features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained includes: Based on the deep neural network unit in the initial model, feature extraction is performed on the material defect image to be trained to obtain implicit features of the material defect image to be trained; wherein the implicit features of the material defect image to be trained represent key information of defects in the material defect image to be trained.

13. A material defect information identification device, characterized in that: include: An acquisition module, used for acquiring data to be identified, wherein the data to be identified includes a material defect image to be identified and at least one scene data; the material defect image to be identified is a material defect image; An extraction module, configured to extract features of the material defect image to be identified according to a recognition model to obtain implicit features of the material defect image to be identified; wherein the recognition model includes at least one scene learner; and based on the scene learner corresponding to the scene data in the recognition model, extract features of the scene data to obtain scene features of the scene data; The recognition module is used to recognize the implicit features and the scene features based on the recognition model to obtain defect information of the material defect image to be recognized.

14. A model training device for defect information recognition, characterized in that: include: An acquisition module, used for acquiring data to be trained, wherein the data to be trained includes a material defect image to be trained and at least one scene data; the material defect image to be trained is a material defect image; An extraction module, configured to extract features of the material defect image to be trained according to the initial model to obtain implicit features of the material defect image to be trained; wherein the initial model includes at least one scene learner; and according to the scene learner corresponding to the scene data in the initial model, extract features of the scene data to obtain scene features of the scene data; A training module is used to train the initial model according to the implicit features and the scene features to obtain a recognition model; wherein the recognition model is used to identify the material defect image to be identified to obtain defect information.

15. An electronic device, characterized in that: The electronic device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1-5 or 6-12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1-5 or 6-12 when executed by a processor.

17. A computer program product, characterized in that The computer program product comprises a computer program, which implements the method according to any one of claims 1 to 5 or 6 to 12 when executed by a processor.

Images