Defect type identification method and model training method, device, and storage medium

Abstract

The application relates to a defect type identification method and device, a model training method and a storage medium. The method comprises the following steps: obtaining a device detection image of a to-be-detected device and device detection information of the to-be-detected device; the device detection image is an image obtained based on the result of photographing the to-be-detected device; the method performs photographing angle identification according to the device detection information and the device detection image in a photographing angle classification layer of a defect type identification model, obtains photographing angle information, and performs defect identification according to the photographing angle information and the device detection image in a defect identification layer of the defect type identification model, and identifies a target defect type corresponding to the to-be-detected device from defect types corresponding to the photographing angle information, that is, the photographing angle of the to-be-detected device is identified first, and then the target defect type of the to-be-detected device is identified from the defect types corresponding to the photographing angle, so that the identification accuracy of a defective product in the defect type identification process of the to-be-detected device is improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a defect type recognition method, model training method, device and storage medium. Background Technology

[0002] In Surface Mount Technology (SMT) production lines, the appearance of electronic components needs to be inspected before and after reflow soldering to confirm the process. Currently, SMT production lines commonly use Automated Optical Inspection (AOI) equipment. The relevant parameters of the components to be inspected are set in the AOI equipment, such as relative position coordinates, length, width, height, offset angle, and color. Then, the components to be inspected are photographed and compared.

[0003] Based on current usage, AOI equipment can intercept most surface mount defects and soldering defects. However, some defective products are still judged as normal and flow to the next process. When the next process discovers new defects, the program parameters need to be adjusted accordingly so that the equipment can intercept the new defects. However, adjusting the judgment parameters may lead to new problems such as misjudgment. Therefore, how to accurately identify defective products in the soldering stage has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a defect type identification method, a model training method, an apparatus, and a storage medium to solve the technical problem of how to accurately identify defective products.

[0005] Firstly, this application provides a defect type identification method, the method comprising:

[0006] Acquire device inspection images and device inspection information of the device under test. The device inspection images are images obtained based on the results of taking pictures of the device under test, and the device inspection information includes the material information and circuit structure file of the device under test.

[0007] The device detection information and the device detection image are input into the shooting angle classification layer of the defect type recognition model to perform shooting angle recognition, so as to obtain the shooting angle information corresponding to the device to be detected.

[0008] The shooting angle information and the device detection image are input into the defect recognition layer of the defect type recognition model, and the target defect type corresponding to the device to be detected is identified from the defect types corresponding to the shooting angle information.

[0009] Optionally, the shooting angle classification layer includes a device modeling layer and a shooting angle recognition layer. The step of inputting the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device to be detected includes:

[0010] The device detection information is input into the device modeling layer to perform 3D modeling, thereby obtaining the model image corresponding to the device to be detected.

[0011] The model image and the device detection image are input into the shooting angle recognition layer to perform shooting angle recognition and obtain the shooting angle information.

[0012] Optionally, the step of inputting the device detection information into the device modeling layer for stereo modeling to obtain the model image corresponding to the device to be detected includes:

[0013] By using the material information in the device detection information, the device packaging information corresponding to the device to be tested can be obtained;

[0014] Based on the preset device standard information, the circuit structure file in the device testing information, and the device packaging information, a three-dimensional image simulation is performed on the device to be tested to generate the model image.

[0015] Optionally, the step of inputting the model image and the device detection image into the shooting angle recognition layer for shooting angle recognition to obtain the shooting angle information includes:

[0016] The model image is adjusted at least once, and the model image after each angle adjustment is matched with the device detection image to obtain an angle matching result;

[0017] If the angle matching result indicates that the model image after angle adjustment matches the device detection image, the current angle information of the model image is output as the shooting angle information.

[0018] Optionally, the step of acquiring the device inspection image of the device under test and the device inspection information of the device under test further includes:

[0019] Acquire the imaging results of the device under test and the device testing information of the device under test;

[0020] Image enhancement is performed on the captured image of the device under test to obtain the detected image of the device.

[0021] Secondly, this application provides a method for training a defect type identification model, the method comprising:

[0022] The device training image of a preset device, the device training information of the preset device, and the defect annotation information corresponding to the device training image are obtained. The device training image is an image obtained based on the result of taking a picture of the preset device. The device training information is the material information and circuit structure file of the preset device.

[0023] The device training information and the device training image are input into the shooting angle classification layer of the model to be trained to identify the shooting angle and obtain the shooting angle training information corresponding to the preset device.

[0024] The shooting angle training information and the device training image are input into the defect recognition layer of the model to be trained, and the defect type training information corresponding to the preset device is identified from the defect type corresponding to the shooting angle training information.

[0025] By comparing the defect type training information and the defect annotation information, the target loss information is obtained;

[0026] Based on the target loss information, the model to be trained is trained to obtain the defect type identification model as described in any of the first aspects.

[0027] Thirdly, this application provides a defect type identification device, the device comprising:

[0028] The first acquisition module is used to acquire a device detection image of the device under test and device detection information of the device under test. The device detection image is an image obtained based on the result of taking a picture of the device under test, and the device detection information is the material information and circuit structure file of the device under test.

[0029] An angle recognition module is used to input the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model to perform shooting angle recognition and obtain the shooting angle information corresponding to the device to be detected.

[0030] The type recognition module is used to input the shooting angle information and the device detection image into the defect recognition layer of the defect type recognition model, and to identify the target defect type corresponding to the device to be detected from the defect types corresponding to the shooting angle information.

[0031] Fourthly, this application provides a defect type identification model training device, the device comprising:

[0032] The second acquisition module is used to acquire a device training image of a preset device, device training information of the preset device, and defect annotation information corresponding to the device training image. The device training image is an image obtained based on the result of taking a picture of the preset device, and the device training information is the material information and circuit structure file of the preset device.

[0033] The first recognition module is used to input the device training information and the device training image into the shooting angle classification layer of the model to be trained to perform shooting angle recognition, so as to obtain the shooting angle training information corresponding to the preset device.

[0034] The second recognition module is used to input the shooting angle training information and the device training image into the defect recognition layer of the model to be trained, and to identify the defect type training information corresponding to the preset device from the defect type corresponding to the shooting angle training information.

[0035] The comparison module is used to compare the defect type training information and the defect annotation information to obtain target loss information;

[0036] The training module is used to train the model to be trained based on the target loss information to obtain a defect type identification model.

[0037] Fifthly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0038] Memory, used to store computer programs;

[0039] When a processor executes a program stored in memory, it implements the defect type identification method described in any embodiment of the first aspect or the defect type identification model training method described in the second aspect.

[0040] In a sixth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the defect type identification method as described in any embodiment of the first aspect or the defect type identification model training method as described in the second aspect.

[0041] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application acquires a device detection image and device detection information of the device to be tested. The device detection image is an image obtained based on the result of taking a picture of the device to be tested, and the device detection information is the material information and circuit structure file of the device to be tested. The device detection information and device detection image are input into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device to be tested. The shooting angle information and device detection image are input into the defect recognition layer of the defect type recognition model to identify the target defect type corresponding to the device to be tested from the defect types corresponding to the shooting angle information. This method can acquire the device inspection image and device inspection information of the device under test. In the shooting angle classification layer of the defect type recognition model, the shooting angle is identified based on the device inspection information and the device inspection image to obtain the shooting angle information. In the defect recognition layer of the defect type recognition model, the target defect type of the device under test is identified from the defect types corresponding to the shooting angle information based on the shooting angle information and the device inspection image. That is, the shooting angle of the device under test is identified first, and then the target defect type of the device under test is identified from the defect types corresponding to the shooting angle. Thus, the defect type can be classified based on spatial features first, and then the defects under the spatial features can be identified further, thereby improving the accuracy of defect product identification in the defect type identification process of the device under test. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0045] Figure 1 A system architecture diagram of a defect type identification method provided in one embodiment of this application;

[0046] Figure 2A flowchart illustrating a defect type identification method provided in one embodiment of this application;

[0047] Figure 3 A flowchart illustrating a defect type identification model training method provided in one embodiment of this application;

[0048] Figure 4 A schematic diagram of a defect type identification device provided in one embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of a defect type recognition model training device provided in one embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0053] To address the technical problem of accurately identifying defective products in the prior art, this application provides a defect type identification method, model training method, apparatus, and storage medium, which can improve the accuracy of defective product identification.

[0054] The first embodiment of this application provides a defect type identification method, which can be applied to, for example... Figure 1 The system architecture shown includes at least an image acquisition module 101 and a recognition module 102, which establish a communication connection.

[0055] Next, based on this system architecture, the defect type identification method will be described in detail, such as... Figure 2 The defect type identification method includes:

[0056] Step 201: Obtain the device inspection image and device inspection information of the device under test. The device inspection image is an image obtained based on the result of taking pictures of the device under test, and the device inspection information is the material information and circuit structure file of the device under test.

[0057] Since some defects can only be captured from specific angles, the device inspection image of the device under test can be multiple inspection images obtained by taking pictures of the device under test from multiple angles, thereby ensuring that the defect is captured in the obtained device inspection image when the device under test has a defect.

[0058] The device to be tested can be a surface-mount component, such as resistors, capacitors, inductors, diodes, transistors, MOSFETs, ball grid array (BGA) components, and multi-pin chips, or a through-hole component, such as through-hole discrete devices, dual in-line package (DIP) chips, and various multi-pin through-hole interfaces. This application does not limit the type of device to be tested. This application can detect common defects in the soldering process of surface-mount components, such as: excess components, missing components, incorrect components, excessive solder, insufficient solder, empty solder joints, cold solder joints, solder rejection, solder bridges, solder cracks, solder adhesion to gold fingers, floating solder, whitening, tombstoning, misalignment, polarity reversal, and foreign matter. It can also detect defects in the soldering process of through-hole components, such as: bridging, excessive solder, insufficient PTH solder filling, white spots, solder spikes, dewetting and non-wetting, board deformation, solder bridging, solder joint voids, overflowing solder, solder balls, blowholes or pinholes, cold solder joints, missing solder, and flux residue.

[0059] The device testing information includes the material information of the device under test and the circuit structure file of the device under test. The material information is the bill of materials (BOM), and the circuit structure file can be the structure file (Gerber) of the PCB board.

[0060] Among them, the device detection image can be generated directly based on the image captured, or it can be generated after image enhancement based on the captured image.

[0061] In one embodiment, the steps of acquiring a device detection image and device detection information of a device under test include: acquiring an image of the device under test and device detection information of the device under test; performing image enhancement on the image of the device under test to improve the clarity of the image, and using the enhanced image as the device detection image.

[0062] In this embodiment, image enhancement of the captured images can improve the clarity of the device detection images and make the identification of defect types more accurate.

[0063] Step 202: Input the device detection information and device detection image into the shooting angle classification layer of the defect type recognition model to perform shooting angle recognition and obtain the shooting angle information corresponding to the device to be detected.

[0064] The defect type identification model can be a two-layer model, such as including a shooting angle classification layer and a defect identification layer. The shooting angle classification layer can first identify the shooting angle information of the device to be inspected corresponding to the device inspection image, and then identify the target defect type in the defect type corresponding to the shooting angle information based on the defect identification layer, thereby improving the accuracy of defect type identification.

[0065] In one embodiment, the shooting angle classification layer includes a device modeling layer and a shooting angle recognition layer.

[0066] The steps of inputting device detection information and device detection images into the shooting angle classification layer of the defect type recognition model to obtain the shooting angle information corresponding to the device under test include: inputting device detection information into the device modeling layer to perform three-dimensional modeling to obtain the model image corresponding to the device under test; and inputting the model image and device detection image into the shooting angle recognition layer to perform shooting angle recognition to obtain the shooting angle information.

[0067] In this embodiment, the device to be tested can be modeled in three dimensions based on the device detection information in the device modeling layer to obtain a model image of the device to be tested, such as a 3D welding image. Then, the 3D welding image can be adjusted to the same angle as the device detection image in the shooting angle recognition layer to achieve shooting angle recognition and obtain shooting angle information.

[0068] The shooting angle information can be the shooting angle when capturing the detection image of the device, or it can be the spatial position representing the shooting angle (for example, it can be three-dimensional coordinate data, such as the coordinate data of the device detection image in a spatial coordinate system or a polar coordinate system). The shooting angle information can include at least one shooting angle or at least one spatial position, without limitation.

[0069] Among the various defect types mentioned above, they can be distinguished according to different shooting angles. For example, excess parts, missing parts, and incorrect parts can be classified as shooting angle information 1, which includes spatial position 1, corresponding to a top-down shooting angle. Insufficient solder, cold solder joints, solder bridges, and solder cracks can be classified as shooting angle information 2, which includes spatial position 2, corresponding to a side-view shooting angle. Non-wetting, solder balls, solder bridging, and cold solder joints can be classified as shooting angle information 3, which includes spatial positions 2 and 3, corresponding to a microscopic shooting angle. Kneeling, pulled-out, pinholes or blowholes, and missing solder can be classified as shooting angle information 4, which can include spatial positions 3 and 1, i.e., a microscopic shooting angle and a top-down shooting angle. All of the above spatial positions can be represented by three-dimensional coordinates or polar coordinates. It should be noted that the classification of shooting angle information 1 to 4 above is only illustrative and can be generalized and classified according to actual needs without limitation.

[0070] The shooting angle classification layer can identify the shooting angle, thereby keeping the simulated image at the same angle as the device inspection image, providing a basis for the identification of defect types.

[0071] In one embodiment, the step of inputting device detection information into the device modeling layer for 3D modeling to obtain a model image corresponding to the device under test includes: obtaining device packaging information corresponding to the device under test through the material information in the device detection information; and performing 3D image simulation of the device under test based on preset device standard information, circuit structure file in the device detection information and device packaging information to generate a model image.

[0072] In this embodiment, the Bill of Materials (BOM) and the Gerber structure file of the PCB can be input into the system. The system can index the corresponding datasheet in the system based on the material information of the device under test on the BOM to obtain the packaging information of the device under test. Then, combined with the Gerber information and device standard information (such as the IPC-A-610 standard), a three-dimensional image simulation of the device under test is performed to construct a three-dimensional image of the welding of the device under test, such as a welding 3D image. The 3D image is adjusted to the same angle as the device test image, and the spatial position corresponding to the angle is recorded (which can be three-dimensional coordinate data, such as spatial coordinate system, polar coordinate system, etc.) as the shooting angle information of the device test image.

[0073] In one embodiment, the step of inputting the model image and the device detection image into the shooting angle recognition layer to obtain shooting angle information includes: adjusting the angle of the model image at least once, and matching the model image after each angle adjustment with the device detection image to obtain an angle matching result; if the angle matching result indicates that the model image after angle adjustment matches the device detection image, outputting the current angle information of the model image as shooting angle information.

[0074] In this embodiment, when performing shooting angle recognition in the shooting angle recognition layer, the angle of the model image can be adjusted. After each angle adjustment, the model image and the device detection image are matched to obtain the angle matching result. If the angle matching result indicates that the model image after angle adjustment matches the device detection image, the current angle information of the model image is output as the shooting angle information. If the angle matching result indicates that they do not match, the angle is adjusted again until they match.

[0075] Step 203: Input the shooting angle information and the device detection image into the defect recognition layer of the defect type recognition model, and identify the target defect type corresponding to the device to be detected from the defect types corresponding to the shooting angle information.

[0076] This method can acquire the device detection image and device detection information of the device under test. In the shooting angle classification layer of the defect type recognition model, the shooting angle is identified based on the device detection information and the device detection image to obtain the shooting angle information. In the defect recognition layer of the defect type recognition model, the target defect type of the device under test is identified from the defect type corresponding to the shooting angle information based on the shooting angle information and the device detection image. That is, the shooting angle of the device under test is identified first, and then the target defect type of the device under test is identified from the defect type corresponding to the shooting angle. This improves the accuracy of defect identification of defective products in the defect type identification process of the device under test. In addition, defects can also be classified by spatial features to balance the defect data identification when increasing the defect identification types, and avoid the model training from having a significant bias due to too much defect data of a certain type.

[0077] In one embodiment, before the step of inputting the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device to be detected, the method further includes: obtaining the defect type recognition model.

[0078] The defect type identification model can be trained based on a training model, which can be a deep learning system, such as a convolutional neural network, without limitation. The training process of the defect type identification model is described in the following section on defect type identification model training methods, and will not be repeated here.

[0079] Based on the same technical concept, the second embodiment of this application provides a method for training a defect type identification model, such as... Figure 3 The methods include:

[0080] Step 301: Obtain the device training image of the preset device, the device training information of the preset device, and the defect annotation information corresponding to the device training image. The device training image is an image obtained based on the result of taking pictures of the preset device, and the device training information is the material information and circuit structure file of the preset device.

[0081] The preset components can be surface-mount components, such as resistors, capacitors, inductors, diodes, transistors, MOSFETs, BGA components, and multi-pin chips, or through-hole components, such as through-hole discrete components, DIP-packaged chips, and various multi-pin through-hole interfaces. Based on the defect type recognition model trained using these preset components, defects in the device to be inspected can be identified.

[0082] The device training images are obtained from photographs of preset devices, including normal and defective images. Defect annotations on the defective images allow for the labeling of the component body, pins, and pads. Labeling the component body identifies its type, labeling all pins allows for the detection of solder joints and solder condition, and labeling the component body and all pad areas reveals the specific shooting angle. During model training, when labeling the soldering status of each pin, both normal and defective pins should be labeled to provide more shooting angle information. The number of defective images corresponding to different shooting angles should be similar or the same.

[0083] Step 302: Input the device training information and device training image into the shooting angle classification layer of the model to be trained to perform shooting angle recognition and obtain the shooting angle training information corresponding to the preset device.

[0084] Step 303: Input the shooting angle training information and device training image into the defect recognition layer of the model to be trained, and identify the defect type training information corresponding to the preset device from the defect type corresponding to the shooting angle training information.

[0085] Step 304: Compare the defect type training information and defect annotation information to obtain the target loss information.

[0086] The target loss information can be a loss function, such as cross-entropy, which can sensitively measure the difference between defect type training information and defect labeling information, thereby improving the performance of the defect type recognition model.

[0087] Step 305: Based on the target loss information, train the model to be trained to obtain the defect type recognition model.

[0088] In this embodiment, the defect type recognition model training method can train the model to be trained based on the device training image of the preset device, the device training information of the preset device, and the defect annotation information corresponding to the device training image. The trained defect type recognition model is a two-layer recognition model including a shooting angle classification layer and a defect recognition layer, which can identify the defect type of the device to be detected.

[0089] In the above embodiments, by training the defect type recognition model and using the defect type recognition model to perform defect recognition, the shooting angle information is first identified during defect recognition, and then the target defect type corresponding to the device under test is identified from the defect types corresponding to the shooting angle information, so that the model can maintain the accuracy of defect recognition when the types of defect recognition increase.

[0090] Furthermore, this defect type recognition model, trained on a large number of images of good products (normal images) and existing images of defective product types (defective images), possesses the ability to autonomously learn new defect types and can identify new defects without adjusting the judgment parameters. During use, the defect type recognition model can add new judgment criteria through learning, and adding new defect judgments does not sacrifice existing criteria. This improves the coverage of defect type recognition and avoids misjudgments. In addition, this defect type recognition model is universal for all products; when the production line produces different products, there is no need to switch the corresponding judgment program, improving the compatibility of defect recognition.

[0091] Based on the same technical concept, the third embodiment of this application provides a defect type identification device, such as... Figure 4 The device includes:

[0092] The first acquisition module 401 is used to acquire a device detection image of the device under test and device detection information of the device under test. The device detection image is an image obtained based on the result of taking a picture of the device under test, and the device detection information is the material information and circuit structure file of the device under test.

[0093] Angle recognition module 402 is used to input the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model to perform shooting angle recognition, so as to obtain the shooting angle information corresponding to the device to be detected.

[0094] The type recognition module 403 is used to input the shooting angle information and the device detection image into the defect recognition layer of the defect type recognition model, and to identify the target defect type corresponding to the device to be detected from the defect types corresponding to the shooting angle information.

[0095] This device can identify defects through a dual-layer recognition model. First, it determines the spatial features of the target device based on the target image at the target angle. Based on the spatial features and the shooting angle classification layer, it determines at least one type of defect. Then, based on the defect recognition layer and the target image, it determines the specific target defect type of the target image from at least one type of defect, thereby improving the accuracy of identifying defective products.

[0096] This application also provides a defect type identification model training device, such as... Figure 5 The device includes:

[0097] The second acquisition module 501 is used to acquire a device training image of a preset device, device training information of the preset device, and defect annotation information corresponding to the device training image. The device training image is an image obtained based on the result of taking a picture of the preset device, and the device training information is the material information and circuit structure file of the preset device.

[0098] The first recognition module 502 is used to input the device training information and the device training image into the shooting angle classification layer of the model to be trained to perform shooting angle recognition and obtain the shooting angle training information corresponding to the preset device.

[0099] The second identification module 503 is used to input the shooting angle training information and the device training image into the defect identification layer of the model to be trained, and to identify the defect type training information corresponding to the preset device from the defect type corresponding to the shooting angle training information.

[0100] The comparison module 504 is used to compare the defect type training information and the defect annotation information to obtain target loss information;

[0101] The training module 505 is used to train the model to be trained based on the target loss information to obtain a defect type recognition model.

[0102] like Figure 6 As shown in the figure, this application provides an electronic device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0103] Memory 113 is used to store computer programs;

[0104] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the defect type identification method or defect type identification model training method provided in any of the foregoing method embodiments.

[0105] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0106] The communication interface is used for communication between the aforementioned terminal and other devices.

[0107] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0108] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0109] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the defect type identification method or defect type identification model training method provided in any of the foregoing method embodiments.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0112] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0113] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. In the description, suffixes such as "module," "part," or "unit" used to denote elements are used solely for illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" may be used interchangeably.

[0114] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A defect type identification method, characterized in that, The method includes: Acquire device inspection images and device inspection information of the device under test. The device inspection images are images obtained based on the results of taking pictures of the device under test, and the device inspection information includes the material information and circuit structure file of the device under test. The device detection information and the device detection image are input into the shooting angle classification layer of the defect type recognition model to perform shooting angle recognition, so as to obtain the shooting angle information corresponding to the device to be detected. The shooting angle information and the device detection image are input into the defect recognition layer of the defect type recognition model, and the target defect type corresponding to the device to be detected is identified from the defect types corresponding to the shooting angle information. The shooting angle classification layer includes a device modeling layer and a shooting angle recognition layer. The step of inputting the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device to be detected includes: The device detection information is input into the device modeling layer to perform 3D modeling, thereby obtaining the model image corresponding to the device to be detected. The model image and the device detection image are input into the shooting angle recognition layer to perform shooting angle recognition and obtain the shooting angle information.

2. The method according to claim 1, characterized in that, The step of inputting the device detection information into the device modeling layer to perform 3D modeling and obtain the model image corresponding to the device to be detected includes: By using the material information in the device detection information, the device packaging information corresponding to the device to be tested can be obtained; Based on the preset device standard information, the circuit structure file in the device testing information, and the device packaging information, a three-dimensional image simulation is performed on the device to be tested to generate the model image.

3. The method according to claim 1, characterized in that, The step of inputting the model image and the device detection image into the shooting angle recognition layer for shooting angle recognition to obtain the shooting angle information includes: The model image is adjusted at least once, and the model image after each angle adjustment is matched with the device detection image to obtain an angle matching result; If the angle matching result indicates that the model image after angle adjustment matches the device detection image, the current angle information of the model image is output as the shooting angle information.

4. The method according to claim 1, characterized in that, The steps of acquiring the device detection image and the device detection information of the device under test include: Acquire the imaging results of the device under test and the device testing information of the device under test; Image enhancement is performed on the captured image of the device under test to obtain the detected image of the device.

5. A method for training a defect type identification model, characterized in that, The method includes: The device training image of a preset device, the device training information of the preset device, and the defect annotation information corresponding to the device training image are obtained. The device training image is an image obtained based on the result of taking a picture of the preset device. The device training information is the material information and circuit structure file of the preset device. The device training information and the device training image are input into the shooting angle classification layer of the model to be trained to identify the shooting angle and obtain the shooting angle training information corresponding to the preset device. The shooting angle training information and the device training image are input into the defect recognition layer of the model to be trained, and the defect type training information corresponding to the preset device is identified from the defect type corresponding to the shooting angle training information. By comparing the defect type training information and the defect annotation information, the target loss information is obtained; Based on the target loss information, the model to be trained is trained to obtain the defect type identification model as described in any one of claims 1 to 4.

6. A defect type identification device, characterized in that, The device includes: The first acquisition module is used to acquire a device detection image of the device under test and device detection information of the device under test. The device detection image is an image obtained based on the result of taking a picture of the device under test, and the device detection information is the material information and circuit structure file of the device under test. An angle recognition module is used to input the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device under test. The shooting angle classification layer includes a device modeling layer and a shooting angle recognition layer. The step of inputting the device detection information and the device detection image into the shooting angle classification layer of the defect type recognition model for shooting angle recognition to obtain the shooting angle information corresponding to the device under test includes: inputting the device detection information into the device modeling layer for 3D modeling to obtain a model image corresponding to the device under test; and inputting the model image and the device detection image into the shooting angle recognition layer for shooting angle recognition to obtain the shooting angle information. The type recognition module is used to input the shooting angle information and the device detection image into the defect recognition layer of the defect type recognition model, and to identify the target defect type corresponding to the device to be detected from the defect types corresponding to the shooting angle information.

7. A defect type recognition model training device, characterized in that, The apparatus for training a defect type identification model according to claim 5 includes: The second acquisition module is used to acquire a device training image of a preset device, device training information of the preset device, and defect annotation information corresponding to the device training image. The device training image is an image obtained based on the result of taking a picture of the preset device, and the device training information is the material information and circuit structure file of the preset device. The first recognition module is used to input the device training information and the device training image into the shooting angle classification layer of the model to be trained to perform shooting angle recognition, so as to obtain the shooting angle training information corresponding to the preset device. The second recognition module is used to input the shooting angle training information and the device training image into the defect recognition layer of the model to be trained, and to identify the defect type training information corresponding to the preset device from the defect type corresponding to the shooting angle training information. The comparison module is used to compare the defect type training information and the defect annotation information to obtain target loss information; The training module is used to train the model to be trained based on the target loss information to obtain a defect type identification model.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the defect type identification method according to any one of claims 1-4 or the defect type identification model training method according to claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defect type identification method as described in any one of claims 1-4 or the defect type identification model training method as described in claim 5.

Images