Element defect detection method

Abstract

The invention discloses an element defect detection method, which is applied to the field of computer vision and is used for improving the accuracy of element defect detection. The method comprises the following steps: determining a first region sub-image corresponding to each element in an actual measurement image through a target detection algorithm; performing feature extraction on each first region sub-image based on a pre-trained feature extraction model, and determining a first feature vector corresponding to each element; obtaining a second feature vector corresponding to each alternative element corresponding to a predetermined target element; respectively determining a first similarity between each first feature vector and each second feature vector; determining the element corresponding to the maximum first similarity as a to-be-detected element corresponding to the target element; and element defect detection is carried out based on the first area sub-image corresponding to the element to be detected, so that the accuracy of element defect detection is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a component defect detection method. Background Art

[0002] In the prior art, when performing component defect detection, the coordinate information of the target component in the standard image is determined, and for the measured image to be detected, the coordinate information of the target component is mapped to the measured image to obtain the area image corresponding to the component to be detected in the measured image. Then, defect detection is performed on the component to be detected based on the area image corresponding to the component to be detected.

[0003] The problem with the prior art is that, in some cases, there are alternative components of different sizes with the same function as the target component. If the alternative components of different sizes are used in the measured image, and the position information of the target component in the standard image is directly mapped to the measured image, the regional image corresponding to the component to be detected in the measured image is obtained. The regional image obtained in this way does not contain the accurate component to be detected. If the size of the alternative component is smaller than that of the target component, the obtained regional image contains background area interference other than the component to be detected. If the size of the alternative component is larger than that of the target component, the obtained regional image cannot contain the complete component to be detected. In both of the above situations, the subsequent component defect detection will be misjudged due to the inaccurate regional image corresponding to the component to be detected. This makes the accuracy of component defect detection low. Summary of the invention

[0004] The present application provides a component defect detection method to solve the problem of low accuracy of component defect detection in the prior art.

[0005] In a first aspect, the present application provides a component defect detection method, the method comprising:

[0006] Determine the first region sub-image corresponding to each component in the measured image by using a target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine the first feature vector corresponding to each component;

[0007] Acquire a second feature vector corresponding to each replacement element corresponding to a predetermined target element; respectively determine a first similarity between each first feature vector and each second feature vector;

[0008] The component corresponding to the largest first similarity is determined as the component to be detected corresponding to the target component; and component defect detection is performed based on the first area sub-image corresponding to the component to be detected.

[0009] The above technical solution has the following advantages or beneficial effects:

[0010] In the present application, the first area sub-image corresponding to each component in the measured image is first detected, and the features of each first area sub-image are extracted based on the feature extraction model to determine the first feature vector corresponding to each component. Then, the second feature vector corresponding to each replacement component corresponding to the predetermined target component is obtained; the first similarity between each first feature vector and each second feature vector is determined; the component in the measured image corresponding to the largest first similarity and the replacement component can be considered to be the same component, so the component in the measured image corresponding to the largest first similarity can be determined as the replacement component corresponding to the target component, that is, the component corresponding to the largest first similarity is determined as the component to be detected corresponding to the target component; component defect detection is performed based on the first area sub-image corresponding to the component to be detected. The present application determines the replacement component corresponding to the target component in the measured image as the component to be detected by feature vector matching, and performs component defect detection based on the first area sub-image corresponding to the component to be detected determined by the target detection algorithm. Compared with the prior art which directly maps the coordinate information of the target element in the standard image to the measured image to obtain the area image corresponding to the element to be detected in the measured image, the first area sub-image corresponding to the element to be detected determined in the present application avoids the problem of inaccurate area image corresponding to the element to be detected due to the use of a substitute element with a size different from that of the target element, thereby improving the accuracy of component defect detection.

[0011] In an optional implementation, the process of predetermining the second feature vectors corresponding to each replacement element corresponding to the target element includes:

[0012] Acquire a second region sub-image corresponding to the target element in the standard image; perform feature extraction on the second region sub-image based on the feature extraction model to obtain a third feature vector of the target element;

[0013] Acquire a fourth feature vector corresponding to each predetermined set of replacement elements; wherein the fourth feature vector is obtained by concatenating the second feature vectors corresponding to each replacement element in the corresponding set of replacement elements;

[0014] Based on the feature extraction model, feature normalization processing is performed on the third feature vector and each fourth feature vector; the second similarity between the third feature vector after feature normalization processing and each fourth feature vector is determined; and the second feature vector corresponding to each alternative element in the alternative element set corresponding to the largest second similarity is determined as the second feature vector corresponding to each alternative element corresponding to the target element.

[0015] The above technical solution has the following advantages or beneficial effects: In the present application, a second area sub-image corresponding to the target element in the standard image is obtained; feature extraction is performed on the second area sub-image based on the feature extraction model to obtain the third feature vector of the target element; fourth feature vectors corresponding to each set of alternative elements are predetermined; the second similarity between the third feature vector after feature standardization and each fourth feature vector is determined; the set of alternative elements corresponding to the largest second similarity can be considered as the set of alternative elements corresponding to the target element; therefore, each alternative element in the set of alternative elements corresponding to the largest second similarity is also the alternative element corresponding to the target element, and therefore the second feature vector corresponding to each alternative element is determined as the second feature vector corresponding to each alternative element corresponding to the target element. The present application determines the second feature vector corresponding to each alternative element corresponding to the target element by means of feature vector matching, which provides a basis for subsequently determining the element to be detected corresponding to the target element and performing defect detection.

[0016] In an optional implementation, before extracting features from each first region sub-image based on the pre-trained feature extraction model, the method further includes:

[0017] Obtain target identification information of the target element, and determine the replacement element set to which the target element belongs in each pre-saved replacement element set; determine whether there is a replacement element with a size different from that of the target element based on size information of each replacement element in the replacement element set, and if so, perform subsequent steps of extracting features from each first area sub-image based on a pre-trained feature extraction model.

[0018] The above-mentioned technical solution has the following advantages or beneficial effects: before performing feature extraction on each first area sub-image based on a pre-trained feature extraction model, the present application first detects whether there are alternative components with a size different from that of the target component. If so, feature extraction is performed on each first area sub-image based on the pre-trained feature extraction model, and a subsequent defect detection process is performed. Among them, the target identification information of the target component is obtained, and the target identification information can be unique identification information such as the main material number of the target component. According to the target identification information, the alternative component set to which the target component belongs in each pre-saved alternative component set can be determined. The size information of each alternative component is also saved in the alternative component set; therefore, according to the size information of each alternative component in the alternative component set, it can be determined whether there are alternative components with a size different from that of the target component.

[0019] In an optional embodiment, if it is determined that there is no substitute component with a size different from that of the target component, the method further includes:

[0020] Acquire the center point coordinate information of the target element, and map the center point coordinate information to the measured image;

[0021] A first region sub-image to which the center point coordinate information in the measured image belongs is determined, and component defect detection is performed based on the first region sub-image.

[0022] The above technical solution has the following advantages or beneficial effects: In the present application, if there is no substitute component with a different size from the target component, it means that the component to be detected and the target component have the same size and appearance. Therefore, the center point coordinate information of the target component is obtained, and the center point coordinate information is mapped to the measured image; the first area sub-image to which the center point coordinate information in the measured image belongs is determined, and the first area sub-image is the component to be detected, and component defect detection is performed based on the first area sub-image to which it belongs. When there is no substitute component with a different size from the target component, the present application can determine the first area sub-image corresponding to the component to be detected according to the coordinates and perform component defect detection without the need for feature matching and other processes. While ensuring the accuracy of defect detection, the algorithm complexity is reduced and the efficiency of component defect detection is improved.

[0023] In an optional implementation, determining the first region sub-image corresponding to each element in the measured image by using a target detection algorithm includes:

[0024] Acquire coordinate information of the target element, and map the coordinate information to the measured image;

[0025] Determine an initial image area corresponding to the coordinate information in the measured image, and expand the initial image area according to a preset area expansion parameter to obtain a target image area;

[0026] The first area sub-image corresponding to each element in the target image area in the measured image is determined by a target detection algorithm.

[0027] The above technical solution has the following advantages or beneficial effects: In the present application, considering that the coordinate information of the component to be detected in the measured image and the coordinate information of the target component in the standard image are within a certain area range, in order to take into account the data processing amount of the algorithm and the accuracy of determining the component to be detected, the coordinate information of the target component in the standard image is obtained, and then the coordinate information is mapped to the measured image; the initial image area corresponding to the coordinate information in the measured image is determined, and then the initial image area is expanded according to the preset area expansion parameters to obtain the target image area; then the first area sub-image corresponding to each component is determined by the target detection algorithm in the target image area in the measured image. Then, the subsequent component defect detection process is continued based on the first area sub-image corresponding to each component in the target image area. Compared with continuing the subsequent component defect detection process for the first area sub-image corresponding to all components in the measured image, the data processing amount of the algorithm is reduced, and the accuracy of determining the components to be detected is also guaranteed.

[0028] In an optional implementation, the training process of the feature extraction model includes:

[0029] For each group of sample component images in a training set, the group of sample component images and the corresponding similarity labels are input into a feature extraction model to be trained; feature extraction is performed on each image in the group of sample component images based on the feature extraction model to obtain each sample feature vector; sample similarities between the each sample feature vectors are determined; a loss value is determined according to the sample similarities and the similarity labels; and the feature extraction model is trained according to the loss value.

[0030] The above technical solution has the following advantages or beneficial effects: In the present application, a feature extraction model is trained based on each group of sample component images in the training set. For each group of sample component images, if two sample component images in the group are images corresponding to two components that replace each other, the similarity label corresponding to the group is 1; if two sample component images in the group are not images corresponding to two components that replace each other, the similarity label corresponding to the group is 0. The group of sample component images and the corresponding similarity labels are input into the feature extraction model to be trained; based on the feature extraction model, the sample feature vectors corresponding to the two images in the group of sample component images are determined; then the sample similarity between the two sample feature vectors is determined; the loss value is determined according to the sample similarity and the similarity label; and the feature extraction model is trained according to the loss value. This provides a basis for subsequently determining the component to be detected corresponding to the target component and performing defect detection.

[0031] In an optional implementation, performing component defect detection based on the first region sub-image corresponding to the component to be detected includes:

[0032] The first area sub-image corresponding to the component to be inspected is input into a pre-trained component defect detection model, and a defect detection result of the component to be inspected is determined based on the component defect detection model; wherein, if the component to be inspected has defects, the defect detection result includes the defect type.

[0033] The above technical solution has the following advantages or beneficial effects: In the present application, based on the pre-trained component defect detection model, the component defect detection is performed on the first area sub-image corresponding to the component to be detected. If the component has a defect, the defect type is determined. The defect type is, for example, broken, sideways, turned over, offset, etc. When training the component defect detection model, the component defect detection model can be trained based on the component sample images in the training set and the corresponding defect labels and defect type labels.

[0034] In an optional implementation, after determining the defect detection result of the component to be detected based on the component defect detection model, the method further includes:

[0035] If the defect detection result of the component to be detected is no defect, inputting the first region sub-image corresponding to the component to be detected into a pre-trained character recognition model, and determining the first character information of the component to be detected based on the character recognition model;

[0036] The second character information of each replacement component corresponding to the target component that is pre-saved is obtained. If the first character information exists in each second character information, it is determined that the defect detection result of the component to be detected is non-defective; if the first character information does not exist in each second character information, the defect detection result of the component to be detected is updated to defective.

[0037] The above technical solution has the following advantages or beneficial effects: In order to further make the defect detection more accurate, if the defect detection result of the component to be detected is determined to be defect-free based on the component defect detection model, the first area sub-image corresponding to the component to be detected is input into the pre-trained character recognition model, and the first character information of the component to be detected is determined based on the character recognition model. When training the character recognition model, the character recognition model can be trained based on the component sample image and the corresponding character content label in the training set. In addition, the second character information of each replacement component corresponding to the pre-saved target component is obtained. If the first character information exists in each second character information, it means that the component to be detected is a replacement component of the target component and can achieve the same function as the target component. At this time, the defect detection result of the component to be detected is still determined to be defect-free; if the first character information does not exist in each second character information, it means that the component to be detected is not a replacement component of the target component. At this time, the component to be detected cannot achieve the same function as the target component, that is, a wrong component has occurred. At this time, the defect detection result of the component to be detected is updated to defective. Thereby further improving the accuracy of defect detection.

[0038] In an optional implementation, before predetermining the fourth feature vectors corresponding to each set of replacement elements, the method further includes:

[0039] Receiving a login instruction of a component defect detection system, and obtaining login information carried in the login instruction; if the login information is verified, displaying a function page of the component defect detection system;

[0040] The substitute component set subpage in the function page displays the second identification information of each substitute component set, and the substitute component subpage in the function page displays the image of the substitute component;

[0041] When an instruction to select a set of alternative components is received, the second identification information of the selected set of alternative components is determined; when an operation instruction to select an image of the alternative component is received, the image of the alternative component is added to the set of alternative components corresponding to the second identification information of the selected set of alternative components.

[0042] The above-mentioned technical solution has the following advantages or beneficial effects: The present application provides a front-end component defect detection system, the system interface of which includes a replacement component set sub-page and a replacement component sub-page. Before predetermining the fourth feature vector corresponding to each replacement component set, first enrich and improve each replacement component set through the front-end component defect detection system. When an instruction to select a replacement component set is received, the second identification information of the selected replacement component set is determined; when an operation instruction to select an image of a replacement component is received, the image of the replacement component is added to the replacement component set corresponding to the second identification information of the selected replacement component set. Then determine the fourth feature vector corresponding to each replacement component set. The convenience of personnel operation is improved by the front-end component defect detection system.

[0043] In an optional implementation, when receiving an operation instruction to select the image of the replacement component, adding the image of the replacement component to the replacement component set corresponding to the second identification information of the selected replacement component set includes:

[0044] When a first operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a temporary substitute component of the current production batch, and the image of the substitute component is added to a temporary subset of the substitute component set corresponding to the second identification information of the selected substitute component set; when a second operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a permanent substitute component, and the image of the substitute component is added to a permanent subset of the substitute component set corresponding to the second identification information of the selected substitute component set.

[0045] The above technical solution has the following advantages or beneficial effects: In the present application, a permanent subset and a temporary subset are set in the substitute component set. The image of the substitute component in the permanent subset does not change with the production batch, and is always the substitute component corresponding to the target component; the image of the substitute component in the temporary subset may change with the production batch. After the production batch changes, it may no longer be the substitute component corresponding to the target component. Therefore, after the production batch changes, the temporary subset needs to be deleted, and the image of the temporary substitute component is reselected through the user's operation to update the temporary subset. When a first operation instruction is received for the user to select the image of the substitute component, the image of the substitute component is added to the temporary subset of the substitute component set corresponding to the second identification information of the selected substitute component set; when a second operation instruction is received for the user to select the image of the substitute component, the image of the substitute component is added to the permanent subset of the substitute component set corresponding to the second identification information of the selected substitute component set. The first operation instruction is, for example, clicking the left button of the mouse, and the second operation instruction is, for example, clicking the right button of the mouse, or the two operation instructions are opposite.

[0046] In an optional implementation, before performing component defect detection based on the first region sub-image corresponding to the component to be detected, the method further includes:

[0047] The component quality inspection sub-page in the function page displays a first regional sub-image corresponding to the component to be inspected; if a regional sub-image adjustment operation instruction is received, the first regional sub-image corresponding to the component to be inspected is adjusted according to the regional sub-image adjustment operation instruction to obtain a third regional sub-image; the first regional sub-image corresponding to the component to be inspected is updated using the third regional sub-image, and component defect detection is performed based on the updated first regional sub-image.

[0048] The above technical solution has the following advantages or beneficial effects: considering that the first regional sub-image corresponding to the component to be detected may be inaccurate, which affects the accuracy of defect detection, in this application, the first regional sub-image corresponding to the component to be detected is displayed on the component quality inspection sub-page in the function page; if the user determines that the first regional sub-image corresponding to the component to be detected needs to be adjusted and optimized, the first regional sub-image is adjusted by dragging the mouse, and the electronic device receives the regional sub-image adjustment operation instruction, and adjusts the first regional sub-image corresponding to the component to be detected according to the regional sub-image adjustment operation instruction to obtain the third regional sub-image; then, the component defect detection is performed based on the third regional sub-image. Thereby, the accuracy of component defect detection is further improved.

[0049] In an optional implementation, after performing component defect detection based on the first region sub-image corresponding to the component to be detected, the method further includes:

[0050] The defect type of the component to be inspected is determined, and the defect type is displayed on a defect display subpage in the function page.

[0051] The above technical solution has the following advantages or beneficial effects: In the present application, the display interface of the component defect detection system includes a defect display subpage, and after the defect type of the component to be detected is determined, the defect type is displayed on the defect display subpage in the function page, so that the user can know the defect type intuitively and improve the user experience.

[0052] In a second aspect, the present application provides a component defect detection device, the device comprising:

[0053] A first determination module is used to determine the first region sub-image corresponding to each component in the measured image through a target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine a first feature vector corresponding to each component;

[0054] A second determination module is used to obtain a second feature vector corresponding to each replacement element corresponding to the predetermined target element; and respectively determine a first similarity between each first feature vector and each second feature vector;

[0055] The defect detection module is used to determine the component corresponding to the largest first similarity as the component to be detected corresponding to the target component; and perform component defect detection based on the first area sub-image corresponding to the component to be detected.

[0056] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0057] Memory, used to store computer programs;

[0058] The processor is used to implement the component defect detection method when executing the program stored in the memory.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the component defect detection method is implemented.

[0060] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the component defect detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 Schematic diagram of the standard images provided for this application;

[0063] Figure 2 It is a flow chart of defect detection in the prior art;

[0064] Figure 3 A schematic diagram of the first component defect detection process provided by this application;

[0065] Figure 4 A schematic diagram of a process for predetermining second feature vectors corresponding to respective replacement elements corresponding to a target element provided in the present application;

[0066] Figure 5A schematic diagram of the second component defect detection provided by this application;

[0067] Figure 6 A schematic diagram of the third component defect detection provided by this application;

[0068] Figure 7 A schematic diagram of a process for determining a first region sub-image corresponding to each element provided in the present application;

[0069] Figure 8 A schematic diagram of the training process of the feature extraction model provided in this application;

[0070] Fig. 9 A schematic diagram of the fourth component defect detection provided by this application;

[0071] Fig.10 A schematic diagram of the component defect detection system interface provided in this application;

[0072] Fig.11 A flow chart for detecting component defects when there is no substitute component with a different size from the target component provided by the present application;

[0073] Fig.12 A flow chart of component defect detection when there is a substitute component with a different size from the target component provided by the present application;

[0074] Fig.13 A schematic diagram of the structure of a component defect detection device provided in this application;

[0075] Fig.14 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0076] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0077] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0078] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0079] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0080] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0082] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0083] Figure 1 This is a schematic diagram of the standard images provided for this application. Figure 2 It is a defect detection flow chart of the prior art, such as Figure 2 As shown, in the prior art, component defect detection is performed by optical detection AOI equipment. The detection method is to first select a PCB board in which the status of each component is normal as a standard PCB board, take a high-definition image of the standard version through an industrial scanning camera, obtain a standard image, and then locate the coordinate information of each component on the PCB board through the process program corresponding to the PCB board. After obtaining the measured image, the coordinate information of the component in the standard image is mapped to the measured image, and the regional image corresponding to the component to be detected in the measured image of the measured PCB board is obtained, and then the difference between the standard image and the measured image is compared to determine whether there is a defect on the measured image. If all the components on the PCB board in the measured image are consistent with the standard image detection, the AOI device determines that the measured PCB board is normal. Otherwise, the AOI device determines that the PCB board in the measured image is abnormal and needs to be manually re-judged.

[0084] Generally, since the process program corresponds to the corresponding PCB board one by one, the positioning coordinates of each component are relatively accurate. However, there are some special cases, such as some components have alternative components with different sizes from standard components, or the actual placement is not strictly calibrated with the coordinates set in the program, or the PCB board is not tightly clamped on the tooling, etc., which causes the component coordinate information set in the program to be inconsistent with the actual situation, which will affect the accuracy of subsequent component detection.

[0085] In order to solve the problem of poor accuracy in component defect detection in the prior art, the present application provides a component defect detection method. Figure 3 The first component defect detection process schematic diagram provided in this application includes the following steps:

[0086] S101: Determine a first region sub-image corresponding to each component in the measured image by using a target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine a first feature vector corresponding to each component;

[0087] S102: Acquire a second feature vector corresponding to each replacement element corresponding to a predetermined target element; respectively determine a first similarity between each first feature vector and each second feature vector;

[0088] S103: Determine the component corresponding to the largest first similarity as the component to be detected corresponding to the target component; and perform component defect detection based on the first region sub-image corresponding to the component to be detected.

[0089] The component defect detection method provided in the present application is applied to an electronic device, which may be a PC, a computer or other device, or a server.

[0090] After the electronic device acquires the measured image, it determines each component in the measured image through the target detection algorithm and obtains the detection frame of each component. For each component, the regional sub-image surrounded by the detection frame of the component in the measured image is used as the first regional sub-image corresponding to the component. In this way, the first regional sub-image corresponding to each component in the measured image can be determined. A pre-trained feature extraction model is deployed in the electronic device, and the first regional sub-image corresponding to each component is respectively input into the feature extraction model. Based on the feature extraction model, feature extraction is performed on each first regional sub-image to determine the first feature vector corresponding to each component. For example, component A corresponds to the first regional sub-image A. Based on the feature extraction model, feature extraction is performed on the first regional sub-image A to obtain the first feature vector A. The first feature vector A is the first feature vector corresponding to component A.

[0091] The electronic device predetermines the second feature vectors corresponding to each substitute element corresponding to the target element. The second feature vectors corresponding to each substitute element corresponding to the target element are obtained by extracting features from the images of each substitute element corresponding to the target element through a feature extraction model. For each element, the first similarity between the first feature vector corresponding to the element and the second feature vector corresponding to each substitute element is calculated respectively. For example, if the number of substitute elements is 30, then for each element, 30 first similarities corresponding to the element can be determined. If the measured image contains 20 elements, a total of 20*30=600 first similarities can be determined. Then the largest first similarity is selected from them. The element corresponding to the largest first similarity is determined as the element to be detected corresponding to the target element. For example, the largest first similarity is the similarity between the first feature vector corresponding to element m in the measured image and the second feature vector corresponding to the substitute element s corresponding to the target element. At this time, the element corresponding to the largest first similarity is determined to be element m, and element m is used as the element to be detected corresponding to the target element. Then, component defect detection is performed based on the first area sub-image corresponding to the element to be detected m. That is, defect detection is performed on the element to be detected.

[0092] The above technical solution has the following advantages or beneficial effects:

[0093] In the present application, the first area sub-image corresponding to each component in the measured image is first detected, and the features of each first area sub-image are extracted based on the feature extraction model to determine the first feature vector corresponding to each component. Then, the second feature vector corresponding to each replacement component corresponding to the predetermined target component is obtained; the first similarity between each first feature vector and each second feature vector is determined; the component in the measured image corresponding to the largest first similarity and the replacement component can be considered to be the same component, so the component in the measured image corresponding to the largest first similarity can be determined as the replacement component corresponding to the target component, that is, the component corresponding to the largest first similarity is determined as the component to be detected corresponding to the target component; component defect detection is performed based on the first area sub-image corresponding to the component to be detected. The present application determines the replacement component corresponding to the target component in the measured image as the component to be detected by feature vector matching, and performs component defect detection based on the first area sub-image corresponding to the component to be detected determined by the target detection algorithm. Compared with the prior art which directly maps the coordinate information of the target element in the standard image to the measured image to obtain the area image corresponding to the element to be detected in the measured image, the first area sub-image corresponding to the element to be detected determined in the present application avoids the problem of inaccurate area image corresponding to the element to be detected due to the use of a substitute element with a size different from that of the target element, thereby improving the accuracy of component defect detection.

[0094] Figure 4 A schematic diagram of a process for predetermining second feature vectors corresponding to each replacement element corresponding to a target element provided in the present application includes the following steps:

[0095] S201: Acquire a second region sub-image corresponding to a target element in a standard image; perform feature extraction on the second region sub-image based on the feature extraction model to obtain a third feature vector of the target element;

[0096] S202: Acquire fourth feature vectors corresponding to each predetermined set of replacement elements; wherein the fourth feature vector is obtained by concatenating the second feature vectors corresponding to each replacement element in the corresponding set of replacement elements;

[0097] S203: Perform feature normalization processing on the third feature vector and each fourth feature vector based on the feature extraction model; determine the second similarity between the third feature vector after feature normalization and each fourth feature vector; determine the second feature vector corresponding to each alternative element in the alternative element set corresponding to the largest second similarity as the second feature vector corresponding to each alternative element corresponding to the target element.

[0098] In the present application, a second regional sub-image corresponding to the target element in the standard image is obtained. Optionally, the target element in the standard image can be an element at the center position in the standard image, or an element at any position in the standard image. Taking the case where the target element is an element at the center position in the standard image as an example, the coordinate information of the target element can be saved in advance, and the image in the area corresponding to the coordinate information of the target element in the standard image can be determined as the second regional sub-image corresponding to the target element. Alternatively, the regional sub-images corresponding to each element in the standard image are determined by a target detection algorithm, and then the second regional sub-image corresponding to the target element at the center position in the standard image is determined. The second regional sub-image is then input into a feature extraction model; feature extraction is performed on the second regional sub-image based on the feature extraction model to obtain a third feature vector of the target element.

[0099] Obtain the fourth feature vector corresponding to each predetermined set of replacement elements; the fourth feature vector is obtained by concatenating the second feature vectors corresponding to each replacement element in the corresponding replacement element set. For each replacement element set, feature extraction is performed on the image of each replacement element in the replacement element set based on the feature extraction model to determine the second feature vector of each replacement element; each second feature vector is concatenated to determine the fourth feature vector of the replacement element set. Optionally, each second feature vector is concatenated by dimension to determine the fourth feature vector of the replacement element set.

[0100] Then, based on the feature extraction model, feature standardization is performed on the third eigenvector and each fourth eigenvector; by performing feature standardization on the third eigenvector and each fourth eigenvector, a third eigenvector and each fourth eigenvector with the same dimension and scale can be obtained. Optionally, feature standardization can be performed on the third eigenvector and each fourth eigenvector through the convolution layer and the pooling layer in the feature extraction model. Then, the second similarity between the third eigenvector after feature standardization and each fourth eigenvector can be determined; the second eigenvector corresponding to each alternative element in the set of alternative elements corresponding to the largest second similarity is determined as the second eigenvector corresponding to each alternative element corresponding to the target element.

[0101] In the present application, a second region sub-image corresponding to the target element in the standard image is obtained; feature extraction is performed on the second region sub-image based on a feature extraction model to obtain a third feature vector of the target element; fourth feature vectors corresponding to each set of alternative elements are predetermined; the second similarity between the third feature vector after feature standardization and each fourth feature vector is determined; the set of alternative elements corresponding to the largest second similarity can be considered as the set of alternative elements corresponding to the target element; therefore, each alternative element in the set of alternative elements corresponding to the largest second similarity is also the alternative element corresponding to the target element, and therefore the second feature vector corresponding to each alternative element is determined as the second feature vector corresponding to each alternative element corresponding to the target element. The present application determines the second feature vectors corresponding to each alternative element corresponding to the target element by means of feature vector matching, which provides a basis for subsequently determining the element to be detected corresponding to the target element and performing defect detection.

[0102] Figure 5 The second component defect detection schematic diagram provided in this application includes the following steps:

[0103] S301: Acquire target identification information of a target component, determine the set of substitute components to which the target component belongs in each pre-saved set of substitute components; determine the first region sub-image corresponding to each component in the measured image by a target detection algorithm if it is determined that there is a substitute component with a size different from that of the target component according to the size information of each substitute component in the set of substitute components; perform feature extraction on each first region sub-image based on a pre-trained feature extraction model to determine the first feature vector corresponding to each component;

[0104] S302: Acquire a second feature vector corresponding to each replacement element corresponding to a predetermined target element; respectively determine a first similarity between each first feature vector and each second feature vector;

[0105] S303: Determine the component corresponding to the largest first similarity as the component to be detected corresponding to the target component; and perform component defect detection based on the first region sub-image corresponding to the component to be detected.

[0106] Before performing feature extraction on each first area sub-image based on a pre-trained feature extraction model, the present application first detects whether there are alternative components with a size different from that of the target component. If so, feature extraction is performed on each first area sub-image based on the pre-trained feature extraction model, and a subsequent defect detection process is performed. Among them, the target identification information of the target component is obtained, and the target identification information can be unique identification information such as the main material number of the target component. According to the target identification information, the alternative component set to which the target component belongs in each pre-saved alternative component set can be determined. The size information of each alternative component is also saved in the alternative component set; therefore, according to the size information of each alternative component in the alternative component set, it can be determined whether there are alternative components with a size different from that of the target component.

[0107] Figure 6 The third component defect detection schematic diagram provided for this application includes the following steps:

[0108] S401: Acquire target identification information of a target component, determine the replacement component set to which the target component belongs in each pre-stored replacement component set; based on size information of each replacement component in the replacement component set, if it is determined that there is no replacement component with a size different from that of the target component, acquire center point coordinate information of the target component, and map the center point coordinate information to the measured image;

[0109] S402: Determine a first region sub-image to which the center point coordinate information in the measured image belongs, and perform component defect detection based on the first region sub-image.

[0110] In the present application, if there is no substitute element with a different size from the target element, it means that the element to be detected and the target element have the same size and appearance. Therefore, the center point coordinate information of the target element is obtained, and the center point coordinate information is mapped to the measured image; the first area sub-image to which the center point coordinate information in the measured image belongs is determined, and the first area sub-image is the element to be detected, and component defect detection is performed based on the first area sub-image to which it belongs. When there is no substitute element with a different size from the target element, the present application can determine the first area sub-image corresponding to the element to be detected according to the coordinates and perform component defect detection without feature matching and other processes. While ensuring the accuracy of defect detection, the algorithm complexity is reduced and the efficiency of component defect detection is improved.

[0111] Figure 7The schematic diagram of the process of determining the first region sub-image corresponding to each element provided in the present application includes the following steps:

[0112] S501: Acquire coordinate information of the target element, and map the coordinate information to the measured image;

[0113] S502: determining an initial image region corresponding to the coordinate information in the measured image, and expanding the initial image region according to a preset region expansion parameter to obtain a target image region;

[0114] S503: Determine, by a target detection algorithm, a first region sub-image corresponding to each component in the target image region in the measured image.

[0115] In the present application, considering that the coordinate information of the component to be detected in the measured image and the coordinate information of the target component in the standard image are within a certain area range, in order to take into account the data processing amount of the algorithm and the accuracy of determining the component to be detected, the coordinate information of the target component in the standard image is obtained, and then the coordinate information is mapped to the measured image; the initial image area corresponding to the coordinate information in the measured image is determined, and then the initial image area is expanded according to the preset area expansion parameters to obtain the target image area; then the first area sub-image corresponding to each component is determined by the target detection algorithm in the target image area in the measured image. Then, the subsequent component defect detection process is continued based on the first area sub-image corresponding to each component in the target image area. Compared with continuing the subsequent component defect detection process for the first area sub-image corresponding to all components in the measured image, the data processing amount of the algorithm is reduced, and the accuracy of determining the components to be detected is also guaranteed.

[0116] Figure 8 The training process diagram of the feature extraction model provided in this application includes the following steps:

[0117] S601: for each group of sample component images in the training set, input the group of sample component images and the corresponding similarity labels into the feature extraction model to be trained;

[0118] S602: extracting features from each image in the group of sample component images based on the feature extraction model to obtain each sample feature vector; and determining sample similarities between each sample feature vector;

[0119] S603: Determine a loss value according to the sample similarity and the similarity label; and train the feature extraction model according to the loss value.

[0120] The above technical solution has the following advantages or beneficial effects: In the present application, a feature extraction model is trained based on each group of sample component images in the training set. For each group of sample component images, if two sample component images in the group are images corresponding to two components that replace each other, the similarity label corresponding to the group is 1; if two sample component images in the group are not images corresponding to two components that replace each other, the similarity label corresponding to the group is 0. The group of sample component images and the corresponding similarity labels are input into the feature extraction model to be trained; based on the feature extraction model, the sample feature vectors corresponding to the two images in the group of sample component images are determined; then the sample similarity between the two sample feature vectors is determined; the loss value is determined according to the sample similarity and the similarity label; and the feature extraction model is trained according to the loss value. This provides a basis for subsequently determining the component to be detected corresponding to the target component and performing defect detection.

[0121] Optionally, when the loss value meets the requirement, or the number of feature extraction model training times reaches a preset threshold, it is determined that the feature extraction model training is completed.

[0122] Optionally, performing component defect detection based on the first region sub-image corresponding to the component to be detected includes:

[0123] The first area sub-image corresponding to the component to be inspected is input into a pre-trained component defect detection model, and a defect detection result of the component to be inspected is determined based on the component defect detection model; wherein, if the component to be inspected has defects, the defect detection result includes the defect type.

[0124] In the present application, based on the pre-trained component defect detection model, the first area sub-image corresponding to the component to be detected is subjected to component defect detection. If the component has a defect, the defect type is determined. The defect type is, for example, broken, sideways, turned over, offset, etc.

[0125] When training the component defect detection model, the component defect detection model can be trained based on the component sample images in the training set and the corresponding defect labels and defect type labels. The component sample images in the training set and the corresponding defect labels and defect type labels are input into the component defect detection model to be trained, and the component defect detection model is used to predict whether the component sample images are defective and the defect type. The loss value is determined based on the prediction results and the defect labels and defect type labels. The component defect detection model is trained based on the loss value. Optionally, when the loss value meets the requirements, or the number of component defect detection model trainings reaches a preset threshold, it is determined that the component defect detection model training is completed.

[0126] After determining the defect detection result of the component to be detected based on the component defect detection model, the method further includes:

[0127] If the defect detection result of the component to be detected is no defect, inputting the first region sub-image corresponding to the component to be detected into a pre-trained character recognition model, and determining the first character information of the component to be detected based on the character recognition model;

[0128] The second character information of each replacement component corresponding to the target component that is pre-saved is obtained. If the first character information exists in each second character information, it is determined that the defect detection result of the component to be detected is non-defective; if the first character information does not exist in each second character information, the defect detection result of the component to be detected is updated to defective.

[0129] In order to further make the defect detection more accurate, if the defect detection result of the component to be detected is determined to be defect-free based on the component defect detection model, the first region sub-image corresponding to the component to be detected is input into the pre-trained character recognition model, and the first character information of the component to be detected is determined based on the character recognition model. When training the character recognition model, the character recognition model can be trained based on the component sample images and the corresponding character content labels in the training set.

[0130] The component sample images and corresponding character content labels in the training set are input into the character recognition model to be trained, and the character content in the component sample images is predicted based on the character recognition model. The loss value is determined based on the predicted character content and the character content label. The character recognition model is trained based on the loss value. Optionally, when the loss value meets the requirements, or the number of character recognition model trainings reaches a preset threshold, it is determined that the character recognition model training is completed.

[0131] In addition, the second character information of each replacement component corresponding to the pre-saved target component is obtained. If the first character information exists in each second character information, it means that the component to be detected is a replacement component of the target component and can achieve the same function as the target component. At this time, the defect detection result of the component to be detected is still determined to be non-defective; if the first character information does not exist in each second character information, it means that the component to be detected is not a replacement component of the target component. At this time, the component to be detected cannot achieve the same function as the target component, that is, a wrong component has occurred. At this time, the defect detection result of the component to be detected is updated to defective. This further improves the accuracy of defect detection.

[0132] Fig. 9 The fourth component defect detection schematic diagram provided in this application includes the following steps:

[0133] S701: Determine the first region sub-image corresponding to each component in the measured image by using a target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine a first feature vector corresponding to each component;

[0134] S702: Obtain a second feature vector corresponding to each replacement element corresponding to a predetermined target element; and respectively determine a first similarity between each first feature vector and each second feature vector;

[0135] S703: Determine the component corresponding to the largest first similarity as the component to be detected corresponding to the target component; input the first region sub-image corresponding to the component to be detected into a pre-trained component defect detection model, and determine the defect detection result of the component to be detected based on the component defect detection model; wherein, if the component to be detected has a defect, the defect detection result includes the defect type;

[0136] S704: If the defect detection result of the component to be inspected is no defect, input the first region sub-image corresponding to the component to be inspected into a pre-trained character recognition model, and determine the first character information of the component to be inspected based on the character recognition model;

[0137] S705: Obtain the pre-saved second character information of each replacement component corresponding to the target component, and if the first character information exists in each second character information, determine that the defect detection result of the component to be detected is non-defective; if the first character information does not exist in each second character information, update the defect detection result of the component to be detected to defective.

[0138] Fig.10 The schematic diagram of the component defect detection system interface provided for the present application includes a replacement component set subpage, a replacement component subpage, a component quality inspection subpage, a defect display subpage, etc. The replacement component set subpage displays the second identification information of the replacement component set 1, the second identification information of the replacement component set 2, ..., the second identification information of the replacement component set n. The replacement component subpage displays the images of each replacement material. The component quality inspection subpage displays the component image of the defect to be detected. The defect display subpage displays each defect type, such as breakage, sideways, flipping, offset, etc. In addition, as Fig.10 As shown, the component defect detection system interface may also include a full-page display subpage, a standard image subpage, and a measured image subpage. The full-page display subpage displays the image of the entire PCB board, the standard image subpage displays the standard image, and the measured image subpage displays the measured image.

[0139] In an optional implementation, before predetermining the fourth feature vectors corresponding to each set of replacement elements, the method further includes:

[0140] Receiving a login instruction of a component defect detection system, and obtaining login information carried in the login instruction; if the login information is verified, displaying a function page of the component defect detection system;

[0141] The substitute component set subpage in the function page displays the second identification information of each substitute component set, and the substitute component subpage in the function page displays the image of the substitute component;

[0142] When an instruction to select a set of alternative components is received, the second identification information of the selected set of alternative components is determined; when an operation instruction to select an image of the alternative component is received, the image of the alternative component is added to the set of alternative components corresponding to the second identification information of the selected set of alternative components.

[0143] The present application provides a front-end component defect detection system, which receives a login instruction of the component defect detection system and obtains the login information carried in the login instruction; if the login information is verified, the function page of the component defect detection system is displayed. Optionally, the login information includes a login username and a login password. The electronic device pre-stores the login username and login password of the authorized user. When the login username and login password carried in the login instruction of the component defect detection system are the same as the login username and login password of the authorized user pre-stored, the login information is verified and the function page of the component defect detection system is displayed.

[0144] The system interface includes a substitute component set subpage and a substitute component subpage. Before predetermining the fourth feature vector corresponding to each substitute component set, first enrich and improve each substitute component set through the front-end component defect detection system. When receiving an instruction to select a substitute component set, determine the second identification information of the selected substitute component set; when receiving an operation instruction to select an image of a substitute component, add the image of the substitute component to the substitute component set corresponding to the second identification information of the selected substitute component set. Then determine the fourth feature vector corresponding to each substitute component set. The front-end component defect detection system improves the convenience of personnel operation.

[0145] In an optional implementation, when receiving an operation instruction to select the image of the replacement component, adding the image of the replacement component to the replacement component set corresponding to the second identification information of the selected replacement component set includes:

[0146] When a first operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a temporary substitute component of the current production batch, and the image of the substitute component is added to a temporary subset of the substitute component set corresponding to the second identification information of the selected substitute component set; when a second operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a permanent substitute component, and the image of the substitute component is added to a permanent subset of the substitute component set corresponding to the second identification information of the selected substitute component set.

[0147] In the present application, a permanent subset and a temporary subset are set in a substitute component set. The image of the substitute component in the permanent subset does not change with the production batch, and is always the substitute component corresponding to the target component; the image of the substitute component in the temporary subset may change with the production batch. After the production batch changes, it may no longer be the substitute component corresponding to the target component. Therefore, after the production batch changes, the temporary subset needs to be deleted, and the image of the temporary substitute component is reselected through the user's operation to update the temporary subset. When a first operation instruction is received for the user to select the image of the substitute component, the image of the substitute component is added to the temporary subset of the substitute component set corresponding to the second identification information of the selected substitute component set; when a second operation instruction is received for the user to select the image of the substitute component, the image of the substitute component is added to the permanent subset of the substitute component set corresponding to the second identification information of the selected substitute component set. The first operation instruction is, for example, clicking the left mouse button, and the second operation instruction is, for example, clicking the right mouse button, or the two operation instructions are opposite.

[0148] In an optional implementation, before performing component defect detection based on the first region sub-image corresponding to the component to be detected, the method further includes:

[0149] The component quality inspection sub-page in the function page displays a first regional sub-image corresponding to the component to be inspected; if a regional sub-image adjustment operation instruction is received, the first regional sub-image corresponding to the component to be inspected is adjusted according to the regional sub-image adjustment operation instruction to obtain a third regional sub-image; the first regional sub-image corresponding to the component to be inspected is updated using the third regional sub-image, and component defect detection is performed based on the updated first regional sub-image.

[0150] Considering that the first regional sub-image corresponding to the component to be detected may be inaccurate, which affects the accuracy of defect detection, in this application, the first regional sub-image corresponding to the component to be detected is displayed on the component quality inspection sub-page in the function page; if the user determines that the first regional sub-image corresponding to the component to be detected needs to be adjusted and optimized, the first regional sub-image is adjusted by dragging the mouse, and the electronic device receives the regional sub-image adjustment operation instruction, and adjusts the first regional sub-image corresponding to the component to be detected according to the regional sub-image adjustment operation instruction to obtain the third regional sub-image; then, the component defect detection is performed based on the third regional sub-image. This further improves the accuracy of component defect detection.

[0151] In an optional implementation, after performing component defect detection based on the first region sub-image corresponding to the component to be detected, the method further includes:

[0152] The defect type of the component to be inspected is determined, and the defect type is displayed on a defect display subpage in the function page.

[0153] In the present application, the display interface of the component defect detection system includes a defect display subpage. After the defect type of the component to be detected is determined, the defect type is displayed on the defect display subpage in the function page. This allows the user to intuitively know the defect type and improve the user experience. Optionally, the defect display subpage displays multiple defect types. After the defect type of the component to be detected is determined, the defect type can be highlighted on the defect display subpage.

[0154] like Fig.10 As shown, when adding a substitute material on the front page, the production line technician will use the left and right mouse buttons to select whether the current material belongs to a temporary substitute component or a permanent substitute component, which is used for data input into the permanent feature database and the temporary feature database respectively. Optionally, after the frame selects the image of the substitute component, use the left mouse button to add it to the collection corresponding to the permanent substitute component, and the right mouse button to add it to the collection of temporary substitute components.

[0155] The results of personnel re-judgment are used as the input of the model. During the personnel re-judgment process, the detection frame of the reported component will be re-checked. If there is a size problem with the detection frame, the personnel can drag the detection frame to modify the component detection frame. The detection frame after personnel re-checking is used as one of the inputs for model fine-tuning. This model refers to the component target detection model. The model is optimized and trained based on the detection frame after personnel re-checking as a new label.

[0156] All components on the PCB are rechecked to see if they are all checked. Each component has a bit number. If all bit numbers are checked, the current PCB is checked. If all components are checked as defect-free, the current PCB is defect-free.

[0157] All the results of the large model correction are used as the final output, and the personnel will make the final review. The coordinate information of the inspection frame adjusted by the personnel in the large model or the previous step is marked in green, and the personnel will change it to yellow after confirming it with the left mouse button on the inspection frame. If the personnel thinks that the inspection frame before adjustment is used for defect detection, the personnel will right-click the mouse to deny it and return to the original positioning result and change the frame to yellow. The original positioning result is the result before the personnel adjustment. The result of this operation determines a position coordinate that is most suitable for the component, helping the large model to optimize the alternative material feature database. This step only further refines the inspection frame.

[0158] The component defect detection method provided in this application has reliable characteristics:

[0159] Verifiability: Through the association between the algorithm and the front-end display page, the interaction with the workers' operation results, the results of the detection frame position feedback from the collection personnel in real time and the information of permanent replacement components or temporary replacement components are used as the input of the large model (feature extraction model) to determine whether a solution similar to the present application is adopted.

[0160] Real-time: This application makes subsequent logical judgments based on the inference results of the feature extraction model. The feature extraction model can process the image to be inferred (measured image) transmitted to the algorithm by the optical inspection AOI device within a given time period, which meets the real-time characteristics of the trustworthy characteristics;

[0161] Reproducibility: The same image can be transmitted to the same batch of models for inference multiple times. The same results can be obtained by inference multiple times in different operating environments (different platforms, different inference times, and different times), which meets the reproducibility feature of the trustworthiness feature.

[0162] Controllability: This application can be turned on or off by the user at any time. The importance and adoption of the inference results can be determined by the user, which is in line with the controllability feature of the trustworthiness feature.

[0163] The present application takes into account that the presence of alternative components of different sizes is the main reason for inaccurate component defect detection. Therefore, the present application analyzes the target component from two aspects: whether there are alternative components of different sizes, and improves the workflow of the Telus AOI equipment.

[0164] If it is determined that there is no alternative component with a size different from that of the target component, obtain the center point coordinate information of the target component and map the center point coordinate information to the measured image; determine the first area sub-image to which the center point coordinate information in the measured image belongs, and perform component defect detection based on the first area sub-image.

[0165] Optional, Fig.11The present invention provides a flow chart for detecting component defects when there is no substitute component with a different size from the target component. Fig.11 As shown, if it is determined that there is no alternative component with a different size from the target component, the coordinate information of the component to be detected in the mxl file of the measured image is obtained. Among them, the coordinate information of the component to be detected in the mxl file of the measured image is the coordinate information of the target component in the pre-saved standard image mapped to the coordinate information in the measured image. Then the target element of the standard image is determined by the target detection algorithm, and then the coordinate information of the target element of the standard image is obtained. It is judged whether the difference between the horizontal and vertical coordinates of the coordinate information of the target element and the coordinate information of the component to be detected is within the preset threshold range. If so, the regional sub-image corresponding to the coordinate information of the component to be detected is directly subjected to the subsequent component defect detection process based on the component defect detection model and the character recognition model; if not, the regional sub-image corresponding to each component in the measured image is detected by the target detection algorithm, and then the center point coordinate information of the target element is mapped to the measured image; the first regional sub-image to which the center point coordinate information in the measured image belongs is determined, and the subsequent component defect detection process based on the component defect detection model and the character recognition model is performed based on the first regional sub-image.

[0166] Fig.12 The present invention provides a flow chart for detecting component defects when there is a substitute component with a different size from the target component. Fig.12 As shown, the present application optimizes large models or multimodal large models with image understanding capabilities, such as the internvl large model, the Monkey large model, the Tongyi Qianwen large model, etc. The large model includes a feature extraction model. Based on the situation where there are alternative components in the application scenario of this application, the maximum number of alternative components can reach 48, and the large model architecture and loss function are optimized. Based on the situation that the alternative components have permanent alternative components and temporary alternative components, the feature vectors extracted from the large model are divided into a permanent feature database and a temporary feature database. The original feature extraction method of the large model for images and prompt words remains unchanged. On this basis, a feature extraction module for the feature database for feature extraction of alternative components is added. First, the images of all alternative components in the alternative material library are input into the image encoder (feature extraction model) for feature learning. Each group of alternative components that can be replaced by each other is a batch, and the information in all alternative material libraries is extracted into the feature database through the image encoder. The feature vector information extracted from the standard image target component is matched with the feature vector in the feature database to calculate the similarity. The matching formula is:

[0167] M=Upsample(max(F i ·A T ));

[0168] Among them, M is the similarity, Upsample is upsampling, max is the maximum value, F i is the feature vector of the i-th group of replacement components, and A is the feature vector extracted from the target component in the standard image. By comparing the feature vector of the target component with the feature training of all feature databases, and selecting the nearest set of feature vectors as the output of this step, the set of replacement components with the highest similarity to the target component can be determined.

[0169] Then, the first regional sub-image corresponding to each component in the measured image is determined by the target detection algorithm; the features of each first regional sub-image are extracted based on the large model to determine the feature vector corresponding to each component; the similarity between the feature vector corresponding to each component and the feature vector corresponding to each alternative component in the alternative component set determined in the previous step is calculated by the above formula; the component in the measured image corresponding to the greatest similarity is determined as the component to be detected corresponding to the target component; and component defect detection is performed based on the regional sub-image corresponding to the component to be detected.

[0170] The training process of the feature extraction model is as follows: First, all the alternative components are grouped according to whether they can replace each other, and a group of alternative components that can replace each other are grouped as a group, and are sent to the feature extraction model to extract the image feature vector of the alternative component. Given a standard image of a target component, the image feature vector of the target component is extracted through the feature extraction model. The feature vector of the standard image is matched with the feature vector of the feature database of the alternative component. If the alternative component can replace the target component, the similarity between the corresponding feature vectors of the two is set to 1, and the similarity between the corresponding feature vectors that cannot be replaced is set to 0. In this way, a measure of the similarity between all databases and a given image is established. When testing, it is possible to determine which group of alternative components is closest to the feature vector of the standard image based on this measure, thereby determining the characteristics of the corresponding set of alternative components of any standard image. Then, the components and their coordinate information in the measured image that are close to the feature vector of the specified alternative component are determined by comparing the feature vectors.

[0171] Fig.13 A schematic diagram of the structure of a component defect detection device provided in this application, the device comprises:

[0172] The first determination module 131 is used to determine the first region sub-image corresponding to each component in the measured image through the target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine the first feature vector corresponding to each component;

[0173] The second determination module 132 is used to obtain the second feature vectors corresponding to each replacement element corresponding to the predetermined target element; and respectively determine the first similarity between each first feature vector and each second feature vector;

[0174] The defect detection module 133 is used to determine the component corresponding to the largest first similarity as the component to be detected corresponding to the target component; and perform component defect detection based on the first region sub-image corresponding to the component to be detected.

[0175] The second determination module 132 is also used to obtain a second area sub-image corresponding to the target element in the standard image; perform feature extraction on the second area sub-image based on the feature extraction model to obtain a third feature vector of the target element; obtain a fourth feature vector corresponding to each predetermined set of alternative elements; wherein the fourth feature vector is obtained by concatenating the second feature vectors corresponding to each alternative element in the corresponding set of alternative elements; perform feature normalization processing on the third feature vector and each fourth feature vector based on the feature extraction model; determine a second similarity between the third feature vector after feature normalization and each fourth feature vector; and determine the second feature vector corresponding to each alternative element in the set of alternative elements corresponding to the largest second similarity as the second feature vector corresponding to each alternative element corresponding to the target element.

[0176] The device also includes:

[0177] The judgment module 134 is used to obtain the target identification information of the target element and determine the alternative component set to which the target element belongs in each pre-saved alternative component set; based on the size information of each alternative component in the alternative component set, it is determined whether there is an alternative component with a size different from that of the target element, and if so, the first determination module 131 is triggered.

[0178] If the judgment result of the judgment module 134 is that there is no alternative component with a size different from that of the target component, the defect detection module 133 is triggered. At this time, the defect detection module 133 is also used to obtain the center point coordinate information of the target component, and map the center point coordinate information to the measured image; determine the first area sub-image to which the center point coordinate information in the measured image belongs, and perform component defect detection based on the first area sub-image.

[0179] The first determination module 131 is specifically used to obtain the coordinate information of the target element and map the coordinate information to the measured image; determine the initial image area corresponding to the coordinate information in the measured image, expand the initial image area according to the preset area expansion parameters, and obtain the target image area; determine the first area sub-image corresponding to each element in the target image area in the measured image through a target detection algorithm.

[0180] The device also includes:

[0181] The training module 135 is used to input the group of sample component images and the corresponding similarity labels into the feature extraction model to be trained for each group of sample component images in the training set; perform feature extraction on each image in the group of sample component images based on the feature extraction model to obtain each sample feature vector; determine the sample similarity between the each sample feature vector; determine the loss value according to the sample similarity and the similarity label; and train the feature extraction model according to the loss value.

[0182] The defect detection module 133 is specifically used to input the first area sub-image corresponding to the component to be detected into a pre-trained component defect detection model, and determine the defect detection result of the component to be detected based on the component defect detection model; wherein, if the component to be detected has defects, the defect detection result includes the defect type.

[0183] The defect detection module 133 is also used to input the first area sub-image corresponding to the component to be detected into a pre-trained character recognition model if the defect detection result of the component to be detected is non-defective, and determine the first character information of the component to be detected based on the character recognition model; obtain the pre-saved second character information of each replacement component corresponding to the target component, if the first character information exists in each second character information, determine that the defect detection result of the component to be detected is non-defective; if the first character information does not exist in each second character information, update the defect detection result of the component to be detected to defective.

[0184] The second determination module 132 is also used to receive a login instruction of the component defect detection system and obtain the login information carried in the login instruction; if the login information is verified, the function page of the component defect detection system is displayed; the alternative component set subpage in the function page displays the second identification information of each alternative component set, and the alternative component subpage in the function page displays the image of the alternative component; when receiving an instruction to select an alternative component set, the second identification information of the selected alternative component set is determined; when receiving an operation instruction to select the image of the alternative component, the image of the alternative component is added to the alternative component set corresponding to the second identification information of the selected alternative component set.

[0185] The second determination module 132 is further used for, upon receiving a first operation instruction for selecting the image of the replacement component, determining that the replacement component is a temporary replacement component of the current production batch, and adding the image of the replacement component to a temporary subset of the replacement component set corresponding to the second identification information of the selected replacement component set; and upon receiving a second operation instruction for selecting the image of the replacement component, determining that the replacement component is a permanent replacement component, and adding the image of the replacement component to a permanent subset of the replacement component set corresponding to the second identification information of the selected replacement component set.

[0186] The defect detection module 133 is also used to display the first regional sub-image corresponding to the component to be inspected in the component quality inspection sub-page in the function page; if a regional sub-image adjustment operation instruction is received, the first regional sub-image corresponding to the component to be inspected is adjusted according to the regional sub-image adjustment operation instruction to obtain a third regional sub-image; the first regional sub-image corresponding to the component to be inspected is updated using the third regional sub-image, and component defect detection is performed based on the updated first regional sub-image.

[0187] The defect detection module 133 is further configured to determine the defect type of the component to be detected, and display the defect type on the defect display subpage in the function page.

[0188] The present application also provides an electronic device, such as Fig.14 As shown, it includes: a processor 141, a communication interface 142, a memory 143 and a communication bus 144, wherein the processor 141, the communication interface 142, and the memory 143 communicate with each other through the communication bus 144;

[0189] The memory 143 stores a computer program, and when the program is executed by the processor 141, the processor 141 executes any one of the above method steps.

[0190] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0191] The communication interface 142 is used for communication between the above electronic device and other devices.

[0192] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0193] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0194] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the component defect detection methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the component defect detection method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0195] The present application also provides a computer storage readable storage medium, wherein the computer readable storage medium stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0196] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0197] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A component defect detection method, characterized in that: The method comprises: Determine the first region sub-image corresponding to each component in the measured image by using a target detection algorithm; extract features from each first region sub-image based on a pre-trained feature extraction model to determine the first feature vector corresponding to each component; Acquire a second feature vector corresponding to each replacement element corresponding to a predetermined target element; respectively determine a first similarity between each first feature vector and each second feature vector; The component corresponding to the largest first similarity is determined as the component to be detected corresponding to the target component; and component defect detection is performed based on the first area sub-image corresponding to the component to be detected.

2. The method according to claim 1, characterized in that The process of predetermining the second feature vectors corresponding to each replacement element corresponding to the target element includes: Acquire a second region sub-image corresponding to the target element in the standard image; perform feature extraction on the second region sub-image based on the feature extraction model to obtain a third feature vector of the target element; Obtaining fourth feature vectors corresponding to each predetermined set of replacement elements; wherein the fourth feature vector is obtained by concatenating the second feature vectors corresponding to each replacement element in the corresponding set of replacement elements; Based on the feature extraction model, feature normalization processing is performed on the third feature vector and each fourth feature vector; the second similarity between the third feature vector after feature normalization processing and each fourth feature vector is determined; and the second feature vector corresponding to each alternative element in the alternative element set corresponding to the largest second similarity is determined as the second feature vector corresponding to each alternative element corresponding to the target element.

3. The method according to claim 1, characterized in that Before extracting features from each first region sub-image based on the pre-trained feature extraction model, the method further includes: Obtain target identification information of the target element, determine the replacement element set to which the target element belongs in each pre-saved replacement element set; determine whether there is a replacement element with a size different from that of the target element according to size information of each replacement element in the replacement element set, and if so, perform a subsequent step of extracting features from each first region sub-image based on a pre-trained feature extraction model; If it is determined that there is no substitute component having a different size than the target component, the method further comprises: Acquire the center point coordinate information of the target element, and map the center point coordinate information to the measured image; A first region sub-image to which the center point coordinate information in the measured image belongs is determined, and component defect detection is performed based on the first region sub-image.

4. The method according to claim 1, characterized in that Determining the first area sub-image corresponding to each element in the measured image by the target detection algorithm includes: Acquire coordinate information of the target element, and map the coordinate information to the measured image; Determine an initial image area corresponding to the coordinate information in the measured image, and expand the initial image area according to a preset area expansion parameter to obtain a target image area; The first area sub-image corresponding to each element in the target image area in the measured image is determined by a target detection algorithm.

5. The method according to claim 1, characterized in that The training process of the feature extraction model includes: For each group of sample component images in a training set, the group of sample component images and the corresponding similarity labels are input into a feature extraction model to be trained; feature extraction is performed on each image in the group of sample component images based on the feature extraction model to obtain each sample feature vector; sample similarities between the each sample feature vectors are determined; a loss value is determined according to the sample similarities and the similarity labels; and the feature extraction model is trained according to the loss value.

6. The method according to claim 1, characterized in that Performing component defect detection based on the first area sub-image corresponding to the component to be detected includes: The first area sub-image corresponding to the component to be inspected is input into a pre-trained component defect detection model, and a defect detection result of the component to be inspected is determined based on the component defect detection model; wherein, if the component to be inspected has defects, the defect detection result includes the defect type.

7. The method according to claim 6, characterized in that After determining the defect detection result of the component to be detected based on the component defect detection model, the method further includes: If the defect detection result of the component to be detected is no defect, inputting the first region sub-image corresponding to the component to be detected into a pre-trained character recognition model, and determining the first character information of the component to be detected based on the character recognition model; The second character information of each replacement component corresponding to the target component that is pre-saved is obtained. If the first character information exists in each second character information, it is determined that the defect detection result of the component to be detected is non-defective; if the first character information does not exist in each second character information, the defect detection result of the component to be detected is updated to defective.

8. The method according to claim 2, characterized in that Before predetermining the fourth feature vectors corresponding to each set of replacement components, the method further includes: Receiving a login instruction of a component defect detection system, and obtaining login information carried in the login instruction; if the login information is verified, displaying a function page of the component defect detection system; The substitute component set subpage in the function page displays the second identification information of each substitute component set, and the substitute component subpage in the function page displays the image of the substitute component; When an instruction to select a set of alternative components is received, the second identification information of the selected set of alternative components is determined; when an operation instruction to select an image of the alternative component is received, the image of the alternative component is added to the set of alternative components corresponding to the second identification information of the selected set of alternative components.

9. The method according to claim 8, characterized in that When receiving an operation instruction to select the image of the replacement component, adding the image of the replacement component to the replacement component set corresponding to the second identification information of the selected replacement component set includes: When a first operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a temporary substitute component of the current production batch, and the image of the substitute component is added to a temporary subset of the substitute component set corresponding to the second identification information of the selected substitute component set; when a second operation instruction for selecting the image of the substitute component is received, it is determined that the substitute component is a permanent substitute component, and the image of the substitute component is added to a permanent subset of the substitute component set corresponding to the second identification information of the selected substitute component set.

10. The method according to claim 8, characterized in that Before performing component defect detection based on the first area sub-image corresponding to the component to be detected, the method further includes: The component quality inspection sub-page in the function page displays a first regional sub-image corresponding to the component to be inspected; if a regional sub-image adjustment operation instruction is received, the first regional sub-image corresponding to the component to be inspected is adjusted according to the regional sub-image adjustment operation instruction to obtain a third regional sub-image; the first regional sub-image corresponding to the component to be inspected is updated using the third regional sub-image, and component defect detection is performed based on the updated first regional sub-image; After performing component defect detection based on the first region sub-image corresponding to the component to be detected, the method further includes: The defect type of the component to be inspected is determined, and the defect type is displayed on a defect display subpage in the function page.

Images