Surface mounting quality detection method and device, electronic equipment and storage medium

Abstract

The invention relates to a mounting quality detection method and device, electronic equipment and a storage medium. The method is applied to a server, and comprises the following steps: receiving an unqualified image uploaded by mounting detection equipment; the unqualified image is used for representing that the mounting quality of the corresponding mounted component is unqualified, and the judgment result of the unqualified mounting quality is obtained by detecting the mounting image obtained by the mounting detection equipment according to the shot mounting image; inputting the received unqualified images into a mounting reinspection model obtained by pre-training, so that the mounting reinspection model outputs reinspection results of the corresponding unqualified images based on mounting feature data acquired from the received unqualified images; wherein the re-inspection result is used for correcting an unqualified image subjected to false inspection into a qualified image so as to represent that the inspection result of the corresponding mounted component is qualified in mounting.

Description

Technical Field

[0001] This disclosure relates to the field of terminal technology, and in particular to a method and apparatus for mounting quality inspection, electronic equipment, and storage medium. Background Technology

[0002] Intelligent manufacturing refers to a production method that utilizes advanced information and communication technologies, automation technologies, and intelligent equipment to intelligently transform and upgrade the manufacturing process.

[0003] Surface mount technology (SMT) is an automated process in the intelligent manufacturing of electronic devices, used to precisely place surface mount components (SMTs) into designated locations on printed circuit boards (PCBs). To improve SMT accuracy, the quality of the SMT must be inspected after the placement process is complete. In the electronics manufacturing industry, this inspection is typically performed by optical inspection equipment. If the inspection result indicates poor SMT quality, the placement process must be repeated.

[0004] With the continuous upgrading of manufacturing processes, components and mounting processes are becoming increasingly refined, which also increases the difficulty of mounting quality inspection. The accuracy of optical inspection equipment can no longer meet the requirements, and false positives frequently occur. To address this, related technologies have proposed technical solutions for re-inspecting the objects being inspected. However, the re-inspection methods in these technologies cannot simultaneously achieve both accuracy and cost control. Summary of the Invention

[0005] This disclosure provides a method and apparatus for mounting quality inspection, an electronic device, and a storage medium, which can improve the accuracy of mounting quality inspection.

[0006] According to a first aspect of this disclosure, a method for inspecting mounting quality is provided, applied to a server, comprising:

[0007] Receive defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the mounting quality of the corresponding mounted components as defective, and the determination result of the defective mounting quality is obtained by the mounting inspection equipment based on the mounted images captured;

[0008] The received defective images are input into a pre-trained mounting re-inspection model, which then outputs the re-inspection results of the corresponding defective images based on the mounting feature data collected from the received defective images.

[0009] The re-inspection results are used to correct falsely detected non-conforming images to conforming images, thus indicating that the test results of the corresponding mounted components are conforming to the mounting criteria.

[0010] According to a second aspect of this disclosure, a mounting quality inspection system is provided, comprising:

[0011] The mounting inspection equipment uses an assembled camera module to capture images of the mounted components and performs mounting quality inspection on the corresponding mounted components based on the captured images, so as to upload the images of the unqualified mounted components to the edge server.

[0012] The edge server inputs the received defective images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding defective images based on the mounting feature data collected from the received defective images.

[0013] The cloud receives the re-inspection results uploaded by the edge server and summarizes the mounting quality inspection results of all mounted components based on the re-inspection results; wherein, the summarized mounting quality inspection results are used to instruct the mounting equipment or the personnel responsible for mounting operations to remount the unqualified mounting components.

[0014] According to a third aspect of this disclosure, a mounting quality inspection device is provided, applied to a server, comprising:

[0015] The receiving unit receives defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the mounting quality of the corresponding mounted components as defective, and the determination result of the defective mounting quality is obtained by the mounting inspection equipment based on the mounted images captured.

[0016] The re-inspection unit inputs the received non-conforming images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding non-conforming images based on the mounting feature data collected from the received non-conforming images.

[0017] The re-inspection results are used to correct falsely detected non-conforming images to conforming images, thus indicating that the test results of the corresponding mounted components are conforming to the mounting criteria.

[0018] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0019] processor;

[0020] Memory used to store processor-executable instructions;

[0021] The processor implements the method as described in the first aspect by running the executable instructions.

[0022] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0023] In the technical solution disclosed herein, a mounting re-inspection model is deployed in the server for re-inspecting mounting quality. Based on this, after the mounting inspection equipment performs mounting quality inspection on the mounted components, it can upload the detected defective images to the server. The server can then input the received defective images into the mounting re-inspection model, which, based on the mounting feature data collected from the defective images, outputs the re-inspection results of the defective images, thereby correcting falsely detected defective images into acceptable images.

[0024] It should be understood that servers, compared to the front-end manufacturing, assembly, and testing equipment in the electronics manufacturing industry, possess far more abundant computing resources and can deploy complex models. Therefore, after deploying the mounting re-inspection model to a server, the server's abundant computing resources can be used to re-inspect non-conforming images, improving the accuracy of the re-inspection and avoiding the problem of poor re-inspection results caused by simple algorithms in front-end equipment, as seen in related technologies. Furthermore, re-inspection using the mounting re-inspection model only consumes a portion of the server's computing resources and incurs no other costs, avoiding the high re-inspection costs associated with manual re-inspection in related technologies. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure of a mounting quality inspection method;

[0027] Figure 2 This is a schematic diagram illustrating an exemplary embodiment of the present disclosure of a mounting quality inspection system;

[0028] Figure 3 This is a schematic diagram illustrating a mounting inspection process according to an exemplary embodiment of this disclosure;

[0029] Figure 4 This is a schematic diagram illustrating an image preprocessing flow according to an exemplary embodiment of this disclosure;

[0030] Figure 5 This is a block diagram illustrating a mounting quality inspection device according to an exemplary embodiment of this disclosure;

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] In the intelligent manufacturing process of electronic devices, after the component mounting operation is completed, the mounted components must be inspected for mounting quality to ensure that the mounting quality is qualified and to avoid electronic device failure due to poor mounting quality.

[0036] In related technologies, AOI (Automatic Optical Inspection) equipment is used to inspect the mounting quality. During the inspection process, the AOI equipment first captures images of the mounted components and then analyzes the images using a pre-set detection algorithm to determine whether the mounting quality of the corresponding components is up to standard. In traditional electronics manufacturing industries where manufacturing processes have not undergone multiple upgrades, this inspection method typically offers high accuracy.

[0037] However, with the continuous upgrading of manufacturing processes, components and mounting processes are becoming increasingly refined. This is mainly reflected in the fact that the size of components is getting smaller and the range of mounting positions is getting smaller. Under these circumstances, although AOI equipment can still efficiently complete the mounting quality inspection of components, its accuracy can no longer meet the requirements, and false detections often occur.

[0038] It is worth noting that misidentifying a qualified product as a non-qualified product will result in it being returned to the previous process for reprocessing, while misidentifying a non-qualified product as a qualified product will cause the non-qualified product to directly enter the market, causing serious consequences. Therefore, when deploying detection algorithms in AOI equipment, technicians will use the most stringent detection standards possible. For example, in a specific algorithm, if a component falls between qualified and non-qualified based on the content of an image, it can be directly judged as a non-qualified placement.

[0039] Under the aforementioned conditions, in the images detected by AOI equipment in related technologies, false positives often occur when components with acceptable mounting quality are identified as having unacceptable mounting quality. While this does not result in defective products entering the market, a large number of acceptable mounted components are returned to the mounting process, severely impacting production efficiency.

[0040] To address these issues, relevant technologies have proposed solutions for re-inspecting non-compliant images detected by AOI (Automated Optical Inspection) equipment. For example, non-compliant images can be manually re-inspected; however, this method is inefficient and costly due to the need for manual labor. Another approach is to pre-install re-inspection algorithms on the industrial control console, but due to limitations in computing resources and heavy workloads, only simple algorithms can be pre-installed, resulting in low efficiency and accuracy. Yet another approach is to deploy an OCR (Optical Character Recognition) model on the front-end device. While OCR models have lower computational resource requirements, they rely on characters on components for detection, making them ineffective or inaccurate when re-inspecting components without characters.

[0041] It is evident that no method for testing mounting quality has yet been proposed in the relevant technologies that balances the cost and efficiency of re-inspection.

[0042] In view of this, this disclosure proposes a method for mounting quality inspection that can balance the cost, accuracy and efficiency of mounting inspection.

[0043] Figure 1 This is a flowchart illustrating a mounting quality inspection method as an exemplary embodiment of the present disclosure. The method is applied to a server. Figure 1 As shown, the method may include the following steps:

[0044] Step 102: Receive the defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the defective mounting quality of the corresponding mounted components, and the determination result of the defective mounting quality is obtained by the mounting inspection equipment based on the mounted images captured.

[0045] In this disclosure, "mounted component" refers to a component after the mounting process has been completed. "Defective image" refers to an image of a mounted component that is deemed defective by the mounting inspection equipment. The mounting inspection equipment can be the AOI equipment described above, or other equipment used for mounting quality inspection based on optical images. For example, the mounting inspection equipment can be a device equipped with a mounting quality inspection algorithm but without a camera module, which can receive images of mounted components captured by an image-capturing device and perform mounting quality inspection based on those images. Of course, this example is merely illustrative; any device capable of performing mounting quality inspection based on images can be used as the mounting inspection equipment in this disclosure. The specific type of mounting inspection equipment can be determined by those skilled in the art based on the actual situation, and this disclosure does not impose any limitations on this.

[0046] It is worth noting that, to improve the accuracy of tile installation quality inspection, the installation inspection equipment can prioritize image enhancement processing on the acquired images of the installed components. For example, it can increase the brightness of underexposed images, increase the contrast of images, and sharpen images. Of course, this example is only illustrative. Whether to perform image enhancement processing on the acquired images, and what method to use for image enhancement, can be determined by those skilled in the art based on actual needs, and this disclosure does not impose any restrictions in this regard.

[0047] As mentioned above, since mounting inspection equipment usually sets relatively stringent inspection standards, it tends to identify qualified products as unqualified products, that is, it often produces "false positive" false detections rather than identifying unqualified products as qualified products. Therefore, after performing mounting quality inspection on mounted components, the mounting inspection equipment in this disclosure can provide the unqualified images corresponding to the unqualified mounted components to the server, so that the server can re-inspect based on the unqualified images to determine whether the mounting quality of the corresponding mounted components is indeed unqualified.

[0048] Step 104: Input the received non-conforming images into the pre-trained mounting re-inspection model, so that the mounting re-inspection model can output the re-inspection results of the corresponding non-conforming images based on the mounting feature data collected from the received non-conforming images; wherein, the re-inspection results are used to correct the falsely detected non-conforming images into qualified images, so as to indicate that the detection results of the corresponding mounted components are mounted qualified.

[0049] In this disclosure, after receiving the non-conforming image uploaded by the mounting inspection equipment, the server can input the non-conforming image into the mounting re-inspection model, so that the mounting re-inspection model can perform mounting quality re-inspection based on the non-conforming image.

[0050] In this disclosure, the mounting re-inspection model deployed on the server can perform mounting quality re-inspection in different ways.

[0051] In one embodiment, since the "whether the mounting quality is qualified or not" is mostly related to "whether the component has shifted relative to the specified position" and "whether the orientation of the component is reversed relative to the specified orientation", the deployed mounting re-inspection model can include two parts: an offset detection module and an orientation detection module. The offset detection module is used to detect the mounting offset of the component in the mounted assembly corresponding to the unqualified image, while the orientation detection module is used to detect the orientation reversal of the component in the mounted assembly corresponding to the unqualified image. If the mounting offset indicates that the component has not shifted and the orientation reversal indicates that the component is not reversed, then it proves that the mounting quality of the corresponding mounted assembly is qualified, and the mounting re-inspection model outputs the re-inspection result that the corresponding unqualified image is a false detection.

[0052] In this embodiment, the offset detection module and the orientation detection module can be executed sequentially. If the former indicates that a component in the mounted assembly has shifted or its orientation is reversed, this is sufficient to prove that the corresponding defective image does not present a false detection problem, and therefore the latter does not need to be executed. This approach reduces the consumption of processing resources and improves re-inspection efficiency.

[0053] In this embodiment, since the difference between a component being offset and not offset is quite obvious in the image, a relatively accurate judgment can usually be made directly based on the image content. Therefore, the offset detection module can directly determine the mounting offset status of the component in the corresponding mounted assembly based on the defective image. Conversely, whether the orientation of a component is reversed is more subtle in the image and usually cannot be judged directly based on the image content. Therefore, mounting feature data can be extracted first, and then the orientation reversal status of the component can be determined based on the mounting feature data.

[0054] Given the aforementioned differences, offset detection, which consumes relatively fewer processing resources, can be performed first, followed by polarity detection. In other words, the received defective image can be input into the offset detection module first, allowing the module to determine the mounting offset of the component in the corresponding mounted assembly based on the received defective image. If the mounting offset indicates a component misalignment, a re-inspection result indicating the input defective image was not a false positive is output. If the mounting offset indicates the component is not misaligned, the defective image is further provided to the orientation detection module, which extracts the mounting feature data of the corresponding mounted assembly and determines the polarity of the component's orientation based on the extracted feature data. If the polarity indicates the component is not polarized, a re-inspection result indicating the input defective image was a false positive is output; if the polarity indicates the component is polarized, a re-inspection result indicating the input defective image was not a false positive is output.

[0055] It should be noted that situations such as "rotation at any angle relative to the specified mounting position" or "displacement relative to the specified mounting position," where the mounting position does not match the specified position, all fall under the category of component offset. "Reverse orientation" refers to the component's orientation being exactly opposite to the specified orientation.

[0056] In another embodiment, since the criteria for determining whether the mounting quality is qualified are not limited to whether the components are offset or polarized, the mounting re-inspection model may not contain multiple modules. Instead, it learns the rules for determining whether the mounting quality is qualified based on the entire model and performs mounting quality re-inspection on the received unqualified images based on the learned rules.

[0057] Of course, the above examples are all illustrative. How to conduct the re-inspection of the mounting quality in the mounting re-inspection model can be determined by those skilled in the art based on the actual situation, and this disclosure does not impose any restrictions on it.

[0058] In this disclosure, the mounting re-inspection model can be trained based on a basic neural network, such as a Transformer network, CNN (Convolutional Neural Networks), or RNN (Recurrent Neural Networks). Alternatively, to improve training efficiency, the mounting re-inspection model can also be trained based on a "large visual model that has already been pre-trained for a general task." In this case, since the model to be trained has already trained some rules for the general task, the number of rules that need to be trained for the mounting quality re-inspection scenario is reduced, significantly improving the model training efficiency and reducing the amount of sample data required for model training. For example, the model to be trained can be a large visual model pre-trained for an image feature extraction task. In this case, the model to be trained has already completed the training of general rules for extracting features from images during pre-training. Therefore, during actual training, the sample data only needs to be trained to: modify the aforementioned general rules into feature extraction rules applicable to the mounting quality re-inspection scenario, and the logic for determining whether a defective image is a false positive based on the features extracted by these rules. Of course, the examples here are merely illustrative. When training the mounting re-inspection model using a pre-trained model, the specific visual large model pre-trained for which general task can be used can be determined by those skilled in the art based on actual needs. For example, the general task can also be a conventional quality inspection task, an image analysis task, etc. This disclosure does not limit this.

[0059] When training a patch re-inspection model based on the aforementioned large-scale visual model, labeled defective images can be pre-acquired as a sample set for model training. The label indicates whether a false detection exists in the corresponding defective image. During training, multiple defective images from the sample set can be input into the large-scale visual model, which extracts patch feature data from the input defective images and outputs the re-inspection results based on this data. After obtaining the re-inspection results for each defective image, each result is compared with its corresponding label, and the matching degree between the results and labels is calculated. If the matching degree does not reach a preset threshold, the large-scale visual model is iteratively corrected until the matching degree reaches the preset threshold. This example is merely illustrative; the specific method used for model training can be determined by those skilled in the art based on actual needs. For instance, a loss function specifically for model training can be set for iterative correction until the loss function converges. This disclosure does not limit this approach.

[0060] In this disclosure, after receiving the re-inspection results, the server can also report the re-inspection results to the cloud, so that the cloud can perform further operations based on the re-inspection results. For example, the cloud can update the mounting re-inspection model based on the received re-inspection results and redeploy the updated mounting re-inspection model to the server; or, for another example, the cloud can feed back the re-inspection results to the mounting equipment to instruct the mounting equipment or the personnel responsible for the mounting operation to avoid performing the re-mounting operation of qualified mounting components corresponding to the falsely detected unqualified images. Of course, the examples here are only illustrative. The specific further operations performed by the cloud based on the reported re-inspection results can be determined by those skilled in the art according to actual needs. For example, the re-inspection results can also be displayed on the public screen of the entire product manufacturing process so that the personnel can judge for themselves whether the model needs to be adjusted and the direction of the model adjustment, or any two or all of the above three examples can be performed simultaneously. This disclosure does not limit this.

[0061] As can be seen from the above technical solution, this disclosure deploys a mounting re-inspection model in the server for re-inspecting mounting quality. Based on this, after the mounting inspection equipment performs mounting quality inspection on the mounted components, it can upload the detected defective images to the server. The server can then input the received defective images into the mounting re-inspection model, which, based on the mounting feature data collected from the defective images, outputs the re-inspection results of the defective images, thereby correcting the defective images into acceptable images.

[0062] It should be understood that, based on considerations such as production efficiency, refined processes, and cost reduction, front-end equipment in intelligent manufacturing typically only performs simple tasks within its assigned processes, with limited computing resources. It can usually only deploy simple algorithms to execute these tasks. Servers, on the other hand, possess abundant computing resources and relatively complete systems, enabling the deployment of more complex and accurate models. Therefore, this disclosure deploys a mounting re-inspection model with feature extraction capabilities to a server. This leverages the server's abundant computing resources to perform more accurate re-inspections on the non-conforming images uploaded after the initial inspection by the mounting inspection equipment, thus avoiding the problem of low re-inspection accuracy caused by using simple algorithms in front-end equipment in related technologies. Secondly, since this disclosure only requires deploying the model on a server to achieve the re-inspection operation, there are no additional manpower or material costs, avoiding the high costs associated with manual re-inspection in related technologies.

[0063] As mentioned above, the re-inspection results can also be uploaded to the cloud. In this case, the mounting inspection equipment, server, and cloud essentially form a cloud-edge-device architecture. It should be noted that the server forming this cloud-edge-device architecture can be called an edge server.

[0064] To this end, this disclosure also proposes a mounting quality inspection system.

[0065] Figure 2 This is a schematic diagram illustrating a mounting quality inspection system as an exemplary embodiment of this disclosure. Figure 2 As shown, the system may include:

[0066] The mounting inspection equipment 21 takes pictures of the mounted components through the assembled camera module, and performs mounting quality inspection on the corresponding mounted components based on the captured images, so as to upload the unqualified images corresponding to the mounted components whose inspection results are unqualified to the edge server 22.

[0067] Edge server 22 inputs the received defective images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding defective images based on the mounting feature data collected from the received defective images.

[0068] The cloud 23 receives the re-inspection results uploaded by the edge server 22, and summarizes the mounting quality inspection results of all mounted components based on the re-inspection results; wherein, the summarized mounting quality inspection results are used to instruct the mounting equipment or the personnel responsible for mounting operations to remount the unqualified mounting components.

[0069] In this embodiment, the mounting inspection device 21, like the AOI device described above, has an image capture function. Therefore, during inspection, it can capture images of the mounted components by calling the camera module, and perform mounting quality inspection on the corresponding mounted components based on the captured images. The inspection results are then uploaded to the edge server 22.

[0070] In this embodiment, an assembly re-inspection model is deployed in the edge server 22. This model outputs a re-inspection result based on the assembly feature data collected from the defective image upon receiving it. As described above, this assembly re-inspection model can be obtained by training a pre-trained model.

[0071] For example, when training a mounting re-inspection model, the DinoV2 model pre-trained based on the Transformer framework can be used as the training model. The DinoV2 model is typically pre-trained based on a general semantic analysis task, and its last fully connected layer outputs a 384-dimensional feature vector to represent the semantic features extracted from the image. Based on this, when training the DinoV2 model using sample images of defective products, since the model already contains general semantic feature extraction rules, it is only necessary to further modify these rules to make them applicable to mounting quality re-inspection. It is worth noting that the mounting re-inspection model can also include a classifier to determine whether a false detection is occurring based on the feature vector extracted by DinoV2. For example, this classifier can be an SVM (Support Vector Machine), which can perform orientation discrimination, offset discrimination, etc. Of course, this example is merely illustrative; the specific training method for the mounting re-inspection model can be determined by those skilled in the art based on actual needs, and this disclosure does not impose any limitations on it.

[0072] It is worth noting that the cloud-edge-device architecture can include multiple edge servers 22, allowing multiple edge servers 22 to simultaneously perform placement quality re-inspection. Similarly, multiple placement inspection devices can also be used to improve inspection efficiency. Furthermore, to enhance re-inspection accuracy, the same non-conforming image can be provided to different edge servers for separate re-inspection, thus forming cross-validation of the re-inspection results and further improving the accuracy of the re-inspection.

[0073] In this implementation, the cloud platform 23 can aggregate the re-inspection results uploaded by the edge server 22, and based on these re-inspection results, aggregate the placement quality inspection results of all mounted components. This information can then be used to instruct the mounting equipment or the personnel responsible for the mounting operation to remount any defective mounted components. Of course, as mentioned above, the cloud platform 23 can also perform operations such as updating the mounting re-inspection model, which will not be elaborated upon here.

[0074] As can be seen from the above description, after forming a cloud-edge-device architecture, multiple parties can collaborate to complete the mounting quality inspection of mounted components. While taking into account both inspection accuracy and cost control, the assembly quality inspection summary results can also be fed back in a timely manner through the cloud, thereby improving production efficiency.

[0075] The following is a detailed description of the mounting and testing process, using a specific embodiment as an example.

[0076] Figure 3 This is a schematic diagram illustrating a mounting inspection process as an exemplary embodiment of this disclosure. Figure 3As shown, the method may include the following steps:

[0077] Step 301: The AOI device takes images of the mounted components.

[0078] In this embodiment, after the equipment responsible for the mounting process completes the mounting of the components, the AOI equipment can take images of the mounted components.

[0079] Step 302: The AOI device preprocesses the captured images.

[0080] In this embodiment, after capturing images of the mounted components, the AOI device can prioritize image preprocessing to improve the accuracy of mounting quality inspection. For example, it can perform preprocessing as follows: Figure 4 As shown, edge cropping, centroid brightness detection, histogram equalization, gamma transformation and other operations are performed in sequence to ensure that the exposure, white balance and other parameters of the image after preprocessing are normal.

[0081] Step 303: The AOI device performs mounting quality inspection on the preprocessed image.

[0082] In this embodiment, after completing the preprocessing operation, the AOI device can perform mounting quality inspection on the preprocessed images and output the inspection results. For example, each image can be labeled as qualified or unqualified to indicate the inspection results of the corresponding mounted components. The unqualified images are uploaded to the edge processor for re-inspection.

[0083] Step 304: The AOI device uploads the non-compliant images to the edge server.

[0084] Step 305: The edge server inputs the defective images into the mounting re-inspection model.

[0085] In this embodiment, the edge server inputs the defective image into the mounting re-inspection model, so that the mounting re-inspection model can extract feature data from the defective image and output the re-inspection result based on the extracted feature data.

[0086] It is worth noting that this embodiment can use the above-described method of "prioritizing offset determination and then orientation determination" for re-inspection. The offset determination is performed by the ResNet18 deep network included in the mounting re-inspection model, while the orientation determination is performed by the DinoV2 model and SVM included in the mounting re-inspection model.

[0087] Step 306: The mounting re-inspection model inputs the defective images into the ResNet18 deep network.

[0088] Step 307: The ResNet18 deep network determines whether the component is offset; if not offset, proceed to step 308; if offset, proceed to step 313.

[0089] In this embodiment, after the mounting re-inspection model inputs the defective image into the ResNet18 deep network, the ResNet18 deep network can identify the image content to determine whether the components contained in the mounted assembly have shifted relative to the specified range. If so, it proves that the mounting quality of the corresponding mounted assembly is unqualified, and the corresponding defective image does not contain false detections. If not, it proves that the mounting quality of the corresponding mounted assembly is at least not unqualified due to the offset of the mounting position. Therefore, the defective image can be further provided to the DinoV2 model for feature extraction, and the SVM can determine whether there is polar reversal based on the extracted features.

[0090] Step 308: Provide the corresponding defective images to the DinoV2 model.

[0091] Step 309: The DinoV2 model extracts features from the non-compliant images.

[0092] In this embodiment, the DinoV2 model can be trained from the pre-trained large visual model described above, and the extracted feature vector can be 384-dimensional.

[0093] Step 310: Provide the extracted features to the SVM.

[0094] In this embodiment, after the DinoV2 model extracts a 384-dimensional feature vector, it provides this vector to the SVM, which then makes a judgment based on the feature vector. It should be understood that the 384-dimensional feature vector can characterize the component's mounting status on the mounted assembly in detail, thus enabling the SVM to make a more accurate judgment based on this feature vector.

[0095] Step 311: SVM performs orientation determination based on the received features; if there is no orientation abnormality, proceed to step 312; if the orientation is abnormal, proceed to step 313.

[0096] In this embodiment, when the component orientation is not reversed, the SVM output result is that there is no abnormality in the direction, thus determining that the re-inspection result of the corresponding unqualified image is a false detection. However, when the component orientation is reversed, the SVM output result is that there is an abnormality in the direction, thus determining that the re-inspection result of the corresponding unqualified image is not a false detection.

[0097] Step 312: Determine that the re-inspection result of the corresponding unqualified image is a false detection.

[0098] Step 313: Determine that the re-inspection result of the corresponding unqualified image is no false detection.

[0099] Step 314: Upload the re-inspection results to the cloud.

[0100] In this embodiment, after receiving the re-inspection results, the edge server can report them to the cloud. The cloud then summarizes and analyzes the results to determine whether it is necessary to iteratively correct the mounting re-inspection model deployed on the edge server. If so, the mounting re-inspection model is iteratively updated based on the analysis results, and the updated model is redeployed to the edge server to update the model.

[0101] Step 315: The cloud platform summarizes and analyzes the re-inspection results.

[0102] Step 316: The cloud platform updates the mounting re-inspection model based on the analysis results.

[0103] Step 317: The cloud will deploy the updated mounting re-inspection model to the edge server.

[0104] In this embodiment, it should be emphasized that the ResNet18 deep network, DinoV2 model, and SVM mentioned above are all part of the mounting re-inspection model. Therefore, the cloud can update the entire mounting re-inspection model or that part when the output of any one of the three is determined to be inaccurate based on the summary analysis. After the update is completed, at least one of the three can be updated and deployed.

[0105] As can be seen from the above technical solution, this embodiment can use the edge server to re-inspect the unqualified images uploaded by the AOI device based on the pre-deployed mounting re-inspection model, avoiding the high cost caused by manual re-inspection in related technologies, as well as the low accuracy of re-inspection caused by simple algorithms in the front-end device.

[0106] In addition, the edge server also uploads the re-inspection results to the cloud. The cloud then summarizes and analyzes the re-inspection results and updates the mounting re-inspection model deployed on the edge server. This enables the entire system to automatically update the mounting re-inspection model, avoiding the problem of low re-inspection accuracy due to untimely model updates.

[0107] Figure 5 This is a block diagram illustrating a mounting quality inspection device according to an exemplary embodiment of this disclosure. (Refer to...) Figure 5 The device includes a receiving unit 501 and a re-inspection unit 502.

[0108] The receiving unit 501 receives the defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the non-conforming mounting quality of the corresponding mounted components, and the determination result of the non-conforming mounting quality is obtained by the mounting inspection equipment based on the mounted images captured.

[0109] The re-inspection unit 502 inputs the received non-conforming images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding non-conforming images based on the mounting feature data collected from the received non-conforming images.

[0110] The re-inspection results are used to correct falsely detected non-conforming images to conforming images, thus indicating that the test results of the corresponding mounted components are conforming to the mounting criteria.

[0111] Optionally, the mounting re-inspection model includes an offset detection module and an orientation detection module;

[0112] The offset detection module is used to detect the offset status of components in the mounted components corresponding to the unqualified images.

[0113] The orientation detection module is used to detect the polarity of the orientation of components in the mounted assembly corresponding to the defective image;

[0114] The mounting re-inspection model outputs a corresponding unqualified image as a false detection re-inspection result when the mounting offset condition indicates that the component is not offset and the orientation reversal condition indicates that the component is not polarized.

[0115] Optionally, the re-inspection unit 502 is further used for:

[0116] The received defective images are first input into the offset detection module, so that the offset detection module can determine the mounting offset status of the components in the corresponding mounted assembly based on the received defective images.

[0117] If the mounting offset condition indicates that the component is misaligned, a re-inspection result is output to indicate that the input defective image was not falsely detected.

[0118] If the mounting offset status indicates that the component is not offset, the defective image is provided to the orientation detection module, which extracts the mounting feature data of the corresponding mounted component and determines the orientation reversal status of the component based on the extracted mounting feature data. If the orientation reversal status indicates that the component is not reversible, a re-inspection result indicating that the input defective image is a false positive is output. If the orientation reversal status indicates that the component is reversible, a re-inspection result indicating that the input defective image is not a false positive is output.

[0119] Optional,

[0120] The model to be trained for training the mounting re-inspection model is a large visual model that has been pre-trained for a general task.

[0121] Optionally, the mounting re-inspection model is trained based on defective sample images carrying labels; the labels are used to indicate whether the corresponding defective images contain false positives or not; the training process of the mounting re-inspection model includes:

[0122] Multiple defective images from the sample set are input into the visual big model, which extracts the mounting feature data of the input defective images and outputs the re-inspection results of the corresponding defective images based on the extracted mounting feature data.

[0123] The re-inspection results of each sample of unqualified images are compared with their respective labels, and the matching degree between the re-inspection results and the labels is calculated. If the matching degree does not reach the preset threshold, the visual large model is iteratively corrected until the matching degree reaches the preset threshold.

[0124] Optional, also includes:

[0125] The reporting unit 503 reports the re-inspection results to the cloud, so that the cloud can update the mounting re-inspection model based on the re-inspection results and redeploy the updated mounting re-inspection model to the server; and / or, reports the re-inspection results to the cloud, so that the cloud can feed back the re-inspection results to the mounting equipment, so as to instruct the mounting equipment or the personnel responsible for mounting operations to avoid performing the re-mounting operation of the qualified mounting components corresponding to the falsely detected unqualified images.

[0126] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0127] Accordingly, this disclosure also provides a mounting quality inspection device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement a mounting quality inspection method as described in any of the above embodiments, for example, the method may include: receiving a non-conforming image uploaded by a mounting inspection device; the non-conforming image is used to characterize the non-conforming mounting quality of the corresponding mounted component, the determination result of the non-conforming mounting quality is obtained by the mounting inspection device based on the captured mounting image; inputting the received non-conforming image into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs a re-inspection result of the corresponding non-conforming image based on mounting feature data collected from the received non-conforming image; wherein the re-inspection result is used to correct falsely detected non-conforming images to qualified images, so as to characterize the detection result of the corresponding mounted component as mounted qualified.

[0128] Accordingly, this disclosure also provides an electronic device, which includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The programs include instructions for implementing the mounting quality inspection method as described in any of the above embodiments. For example, the method may include: receiving a non-conforming image uploaded by a mounting inspection device; the non-conforming image is used to characterize the non-conforming mounting quality of the corresponding mounted component, and the determination result of the non-conforming mounting quality is obtained by the mounting inspection device based on the captured mounting image; inputting the received non-conforming image into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs a re-inspection result of the corresponding non-conforming image based on mounting feature data collected from the received non-conforming image; wherein the re-inspection result is used to correct falsely detected non-conforming images to qualified images, so as to characterize the detection result of the corresponding mounted component as qualified.

[0129] Figure 6 This is a block diagram illustrating an apparatus 600 for implementing a mounting quality inspection method according to an exemplary embodiment. For example, apparatus 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0130] Reference Figure 6 The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0131] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0132] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of such data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0133] Power supply component 606 provides power to the various components of device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600.

[0134] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0135] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0136] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0137] Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0138] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (New Radio), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0139] In an exemplary embodiment, the apparatus 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0140] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of the device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0141] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0142] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0143] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for inspecting mounting quality, characterized in that, Applied to servers, including: Receive defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the mounting quality of the corresponding mounted components as defective, and the determination result of the defective mounting quality is obtained by the mounting inspection equipment based on the mounted images captured; The received defective images are input into a pre-trained mounting re-inspection model, which then outputs the re-inspection results of the corresponding defective images based on the mounting feature data collected from the received defective images. The re-inspection results are used to correct falsely detected non-conforming images to conforming images, thus indicating that the test results of the corresponding mounted components are conforming to the mounting criteria.

2. The method according to claim 1, characterized in that, The mounting re-inspection model includes an offset detection module and an orientation detection module; The offset detection module is used to detect the offset status of components in the mounted components corresponding to the unqualified images. The orientation detection module is used to detect the polarity of the orientation of components in the mounted assembly corresponding to the defective image; The mounting re-inspection model outputs a corresponding unqualified image as a false detection re-inspection result when the mounting offset condition indicates that the component is not offset and the orientation reversal condition indicates that the component is not polarized.

3. The method according to claim 2, characterized in that, The step of outputting the re-inspection result of the corresponding non-conforming image based on the mounting feature data collected from the received non-conforming image includes: The received defective images are first input into the offset detection module, so that the offset detection module can determine the mounting offset status of the components in the corresponding mounted assembly based on the received defective images. If the mounting offset condition indicates that the component is misaligned, a re-inspection result is output to indicate that the input defective image was not falsely detected. If the mounting offset status indicates that the component is not offset, the defective image is provided to the orientation detection module, which extracts the mounting feature data of the corresponding mounted component and determines the orientation reversal status of the component based on the extracted mounting feature data. If the orientation reversal status indicates that the component is not reversible, a re-inspection result indicating that the input defective image is a false positive is output. If the orientation reversal status indicates that the component is reversible, a re-inspection result indicating that the input defective image is not a false positive is output.

4. The method according to claim 1, characterized in that, The model to be trained for training the mounting re-inspection model is a large visual model that has been pre-trained for a general task.

5. The method according to claim 4, characterized in that, The mounting re-inspection model is trained based on defective sample images carrying labels; the labels are used to indicate whether the corresponding defective images contain false positives or not; the training process of the mounting re-inspection model includes: Multiple defective images from the sample set are input into the visual big model, which extracts the mounting feature data of the input defective images and outputs the re-inspection results of the corresponding defective images based on the extracted mounting feature data. The re-inspection results of each sample of unqualified images are compared with their respective labels, and the matching degree between the re-inspection results and the labels is calculated. If the matching degree does not reach the preset threshold, the visual large model is iteratively corrected until the matching degree reaches the preset threshold.

6. The method according to claim 1, characterized in that, Also includes: The re-inspection results are reported to the cloud, so that the cloud can update the mounting re-inspection model based on the re-inspection results, and redeploy the updated mounting re-inspection model to the server; and / or, The re-inspection results are reported to the cloud, which then feeds back the results to the mounting equipment to instruct the mounting equipment or the personnel responsible for mounting operations to avoid performing the re-mounting operation of qualified mounting components corresponding to the falsely detected unqualified images.

7. A mounting quality inspection system, characterized in that, include: The mounting inspection equipment uses an assembled camera module to capture images of the mounted components and performs mounting quality inspection on the corresponding mounted components based on the captured images, so as to upload the images of the unqualified mounted components to the edge server. The edge server inputs the received defective images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding defective images based on the mounting feature data collected from the received defective images. The cloud receives the re-inspection results uploaded by the edge server and summarizes the placement quality inspection results of all mounted components based on the re-inspection results; the summarized placement quality inspection results are used to instruct the mounting equipment or the personnel responsible for the mounting operation to re-mount the unqualified mounting components.

8. A device for inspecting mounting quality, characterized in that, Applied to servers, including: The receiving unit receives defective images uploaded by the mounting inspection equipment; the defective images are used to characterize the mounting quality of the corresponding mounted components as defective, and the determination result of the defective mounting quality is obtained by the mounting inspection equipment based on the mounted images captured. The re-inspection unit inputs the received non-conforming images into a pre-trained mounting re-inspection model, so that the mounting re-inspection model outputs the re-inspection results of the corresponding non-conforming images based on the mounting feature data collected from the received non-conforming images. The re-inspection results are used to correct falsely detected non-conforming images to conforming images, thus indicating that the test results of the corresponding mounted components are conforming to the mounting criteria.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-6 by executing the executable instructions.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images