Electronic component assembly detection method, electronic device, and storage medium
By extracting and aligning features from assembly images and text using a detection model, the problem of low accuracy in electronic component detection under complex environments is solved, achieving high-precision assembly detection and self-optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG HUARAY TECH CO LTD
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-12
AI Technical Summary
Existing electronic component assembly and testing methods have low accuracy in complex environments and are difficult to distinguish between similar electronic components.
A detection model is used to extract features from component assembly images and specification text. By aligning visual and textual features and combining them with the CLIP network, fine-grained attribute classification capabilities are enhanced, thereby achieving multimodal feature fusion.
It improves the accuracy of electronic component assembly and inspection, can accurately distinguish complex components, reduce the workload of manual verification, and continuously improves inspection accuracy through self-optimization.
Smart Images
Figure CN122199989A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of assembly and testing technology, and in particular to a method for assembling and testing electronic components, as well as electronic devices and storage media. Background Technology
[0002] Currently, template matching algorithms and geometric grayscale information are commonly used to inspect the assembly of electronic components on PCBs (Printed Circuit Boards). Although these methods can ignore changes in the appearance of electronic components caused by scaling, rotation, or posture deformation, they are not easy to distinguish complex electronic components using feature similarity, resulting in low accuracy in the assembly inspection of electronic components. Summary of the Invention
[0003] This application provides at least one method for assembling and inspecting electronic components, as well as an electronic device and a storage medium, which can improve the accuracy of assembly and inspection of electronic components.
[0004] The first aspect of this application provides an electronic component assembly inspection method, which includes: extracting features from a component assembly image using a detection model to obtain component image features, and extracting features from component specification text to obtain component text features corresponding to each electronic component, wherein the component assembly image includes multiple electronic components; aligning the component image features and component text features using a detection model to obtain alignment results corresponding to the electronic components; and analyzing the alignment results to determine the assembly error level of the electronic components in the component assembly image.
[0005] The alignment results include a first alignment result and a second alignment result. The detection model is used to perform feature alignment on the component image features and component text features to obtain the alignment result corresponding to the electronic component. This includes: performing coordinate alignment on the target electronic component in the component assembly image based on the component image features and component text features to obtain the first alignment result; and performing category alignment on the target electronic component based on the component image features and component text features to obtain the second alignment result.
[0006] Specifically, based on component image features and component text features, the target electronic component in the component assembly image is aligned to obtain a first alignment result, including: predicting the target electronic component based on component text features to obtain the predicted coordinates, and performing target detection based on component image features to obtain the position coordinates of the target electronic component; and obtaining the first alignment result based on the deviation between the predicted coordinates and the position coordinates.
[0007] The first alignment result is obtained based on the deviation between the predicted coordinates and the position coordinates, including: if the deviation between the predicted coordinates and the position coordinates is less than or equal to a preset deviation threshold, the first alignment result is successful; or if the deviation between the predicted coordinates and the position coordinates is greater than the preset deviation threshold, the first alignment result is unsuccessful.
[0008] Specifically, based on component image features and component text features, the target electronic components are aligned by category to obtain a second alignment result, which includes: performing target detection on component image features to determine the component category of the target electronic components in the component assembly image; filtering component text features based on component category to obtain several target component text features; and determining the second alignment result based on the similarity between component image features and target component text features.
[0009] The component specification text includes the component list and process specification document corresponding to the component assembly image. Feature extraction is performed on the component specification text to obtain the component text features corresponding to each electronic component. This includes: extracting features based on the component list and process specification document to obtain the component text features corresponding to each electronic component; after analyzing the alignment results to determine the assembly error level of the electronic components in the component assembly image, this includes: determining the prompting method based on the assembly error level.
[0010] The process includes: after using the detection model to align the component image features and component text features to obtain the alignment results corresponding to the electronic components, the following steps are taken: analyzing the alignment results to obtain the analysis results; in response to the analysis results indicating alignment anomalies, regenerating the standard text data for the components with alignment anomalies in the component assembly image, and retraining the detection model using the electronic components with alignment anomalies and the corresponding standard text data.
[0011] The training method for the detection model is as follows: The detection model is used to extract features from the sample component assembly image to obtain the sample component image features corresponding to each sample electronic component, and to extract features from the sample component specification text to obtain the sample component text features corresponding to each sample electronic component. Each sample component assembly image includes at least two sample electronic components. The detection model is used to align the coordinates of the sample electronic components in the component image features and component text features to obtain a first sample alignment result. Furthermore, the detection model is used to align the categories of the sample electronic components in the sample component image features and sample component text features to obtain a second sample alignment result. Based on the deviation between the first sample alignment result and the first labeled data, and the deviation between the second sample alignment result and the second labeled data, the network parameters of the detection model are adjusted until the detection model converges.
[0012] The second aspect of this application provides an electronic device including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the electronic component assembly and inspection method of the first aspect described above.
[0013] A third aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the electronic component assembly and inspection method described in the first aspect above.
[0014] The above scheme utilizes a detection model to extract features from component assembly images, obtaining component image features, and also extracts features from component specification text, obtaining corresponding component text features for each electronic component. The detection model then aligns the component image features and component text features to obtain an alignment result that integrates visual and textual information for the electronic components. The alignment result is then analyzed to determine the assembly error level of the electronic components in the assembly image. This multimodal feature alignment structure enables electronic component assembly detection to integrate visual and textual information, thereby improving the accuracy of distinguishing complex components.
[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0017] Figure 1 This is a schematic flowchart of an embodiment of the electronic component assembly and inspection method of this application; Figure 2 This is a flowchart illustrating an embodiment of the detection model training method of this application; Figure 3 This is a schematic flowchart of another embodiment of the electronic component assembly and inspection method of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic component assembly and testing device of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0020] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0021] While machine vision has made incredible progress, assembly verification and inspection of electronic components remains one of the most challenging areas to automate. Although template matching algorithms can ignore changes in the appearance of electronic components caused by scaling, rotation, or pose distortion, the configuration variations among different electronic components make maintaining traditional algorithms increasingly difficult. This is because it involves multiple variable factors such as lighting, color, curvature, perspective, and field of view. Furthermore, the differences and deviations between very similar electronic components are small, and conditions such as complex and chaotic surface textures and poor lighting still pose significant challenges to assembly inspection. To address these issues, this solution employs... Figure 1 The method described.
[0022] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the electronic component assembly and inspection method of this application. Specifically, it may include the following steps: Step S110: Use the detection model to extract features from the component assembly image to obtain component image features, and extract features from the component specification text to obtain the component text features corresponding to each electronic component. The component assembly image includes multiple electronic components.
[0023] This application primarily relates to the field of assembly inspection of electronic components (i.e., electronic devices such as capacitors, resistors, diodes, etc.), specifically involving image processing and deep learning. By employing a detection model, visual and textual features of electronic components can be fused. Furthermore, by modifying the CLIP (Contrastive Language–Image Pre-training, a neural network connecting text and images) network in the detection model, fine-grained attribute classification capabilities are enhanced, resulting in stronger non-linear representation capabilities of the features output by the detection model.
[0024] The detection model comprises a visual branch and a text branch. The visual branch processes component assembly images and extracts corresponding component image features, while the text branch processes component specification text and extracts corresponding component text features. The component assembly images contain visual information of multiple electronic components, while the component specification text includes attribute descriptions such as component category (e.g., capacitor, diode), size, color, and orientation. The detection model achieves feature alignment through an optimized CLIP architecture. This architecture adds a lightweight attribute classification expert layer (Property Layer) to the classic CLIP image encoder, enabling dynamic fusion of component image features and component text features at a fine-grained attribute level. The component assembly images are optical images of the PCB board captured by a camera.
[0025] By employing a multimodal feature alignment structure, electronic component assembly inspection can integrate visual and textual information, thereby improving the accuracy of distinguishing complex components. Through fine-grained attribute expert layer design, the model can capture the differences in components such as size and color, thus achieving accurate differentiation of similar components.
[0026] In one implementation, a pre-trained CNN (Convolutional Neural Network) model (such as ResNet or EfficientNet) from the field of computer vision can be used in the vision branch. After removing the classification head, feature extraction is performed on the component assembly image, and the output high-level features are the component image features.
[0027] In the text branch, pre-trained language models from the field of natural language processing (such as BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language representation model based on the Transformer structure) and Sentence-BERT) can be used to convert component specification text into fixed-dimensional semantic vectors (i.e., component text features). Specifically, component specification text includes component lists and process specification documents corresponding to component assembly images. To obtain component text features, feature extraction can be performed on the component lists and process specification documents to obtain the component text features corresponding to each electronic component.
[0028] Understandably, no specific limitations are made here regarding the methods for feature extraction from component assembly images and component specification text.
[0029] Step S120: Use the detection model to perform feature alignment on the component image features and component text features to obtain the alignment result corresponding to the electronic component.
[0030] In one embodiment, to improve the accuracy of alignment, coarse positioning and fine positioning can be used to obtain the position information of relevant electronic components in the image and text to be judged, and the corresponding alignment results include a first alignment sub-result and a second alignment sub-result.
[0031] First, based on component image features and component text features, the target electronic components in the component assembly image are aligned to obtain the first alignment result. Specifically, prediction is performed based on component text features to obtain the predicted coordinates of the target electronic components, and target detection is performed on component image features to obtain the position coordinates of the target electronic components. The component text features are processed by a pre-trained text model to generate predicted coordinates, which are based on the CAD (Computer-Aided Design) design positions described in the process specification document. The component image features are processed by a lightweight YOLO-NAS target detection model to output the actual position coordinates. This model can identify the bounding box coordinates of electronic components in the image. Then, based on the deviation between the predicted coordinates and the position coordinates, the first alignment result is obtained.
[0032] If the deviation between the predicted coordinates and the position coordinates is less than or equal to the preset deviation threshold, the first alignment result is considered successful. Alternatively, if the deviation is greater than the preset deviation threshold, the first alignment result is considered a failure. The preset deviation threshold is a system-defined upper limit for coordinate deviation, for example, 1 pixel. The deviation between the predicted and position coordinates is obtained by calculating the Euclidean distance between the two coordinate points, and the system compares this deviation value with the preset threshold in real time. Furthermore, the preset deviation threshold can be set to 0.5 pixels, 1 pixel, or 2 pixels to accommodate different component accuracy requirements; the threshold can be dynamically adjusted based on image acquisition quality or component size; a 1-pixel threshold can be set for capacitive components, and a 0.5-pixel threshold for micro-resistive components, to achieve finer matching control. Through the preset deviation threshold structure, the alignment result determination is based on objective numerical standards, thereby achieving automation and consistency in the detection process; through deviation comparison logic, the system can effectively filter high-error matches, thereby improving the accuracy and reliability of assembly inspection and avoiding misjudgments caused by difficulties in distinguishing feature similarities.
[0033] Secondly, based on the component image features and component text features, the target electronic components are aligned by category to obtain a second alignment result. Specifically, target detection is performed on the component image features to determine the component category of the target electronic components in the component assembly image. Based on the component category, the component text features are filtered to obtain several target component text features, for example, for the capacitor category, text descriptions of blue color and 0-degree orientation are filtered out. Then, based on the similarity between the component image features and the target component text features, the second alignment result is determined, for example, by matching using cosine similarity calculation.
[0034] In one specific implementation, the visual branch of the detection model processes the image features of the components to obtain spatial location information, while the text branch processes the text features of the components to obtain attribute description information. A coordinate alignment operation predicts the text features of the components to determine the predicted coordinates of the target electronic component in the CAD coordinate system. Then, the predicted coordinates of the target electronic component in the CAD coordinate system are compared with the position coordinates of the target electronic component. If the deviation between the predicted coordinates and the position coordinates is less than or equal to a preset deviation threshold, the first alignment result is considered successful, thus achieving coarse positioning.
[0035] Secondly, the category alignment operation calculates the matching relationship between component image features and component text features through a cross-attention mechanism. First, the electronic components in the component image features are classified by using the CLIP model of a multi-expert network. Then, the attribute expert layer is used to extract fine-grained features such as color and orientation of electronic components in the component image features and match them with the component text features.
[0036] Coordinate alignment ensures that the position of the target electronic component in the image is consistent with the design coordinates, while category alignment ensures that the component type matches the text specification description. The two work together in the detection model to first complete the accurate coordinate positioning and then perform category recognition, avoiding the influence of position deviation on category judgment.
[0037] By employing a coordinate alignment structure, the spatial location matching of target electronic components in the image becomes more precise, thereby achieving accurate detection of assembly positions. A category alignment structure enables more accurate identification of component category attributes, thus achieving precise differentiation of similar components. The separate processing mechanism of coordinate alignment and category alignment allows the detection process to independently optimize position and attribute information, thereby improving the robustness of detection in complex component assembly scenarios.
[0038] In one embodiment, after obtaining the alignment results corresponding to the electronic components, the alignment results can be analyzed to obtain analysis results. In response to the analysis result indicating alignment anomalies, the standardized text data of the misaligned components in the component assembly image is regenerated, and the detection model is retrained using the misaligned electronic components and the corresponding standardized text data.
[0039] When the similarity between the component image features and the component text features is lower than a preset similarity threshold, it is identified as an alignment anomaly. In response to the alignment anomaly, the system automatically performs semantic analysis on the abnormal electronic components in the component assembly image, generating structured, standardized text data corresponding to the abnormal electronic component through prompt word engineering. For example, if the capacitor orientation is described incorrectly, the system regenerates the correct text description, "The capacitor orientation should be 0 degrees." Alternatively, templates such as "size=1206, color=blue" can be used to define newly generated features. The detection model is then fine-tuned using the newly generated standardized text data and the abnormal component image. This process requires no manual intervention, automatically expands the feature library, and improves the model's ability to distinguish similar components.
[0040] By automatically analyzing alignment results, the system can instantly identify assembly errors, thereby reducing the workload of manual review. By automatically generating standardized text data, the model can update its feature library based on real-time anomaly data, thus avoiding reliance on external sample annotations. By retraining the model using anomaly data, the detection accuracy continuously improves over time, achieving self-optimization of the assembly inspection system.
[0041] Step S130: Analyze the alignment results to determine the assembly error level of electronic components in the component assembly image.
[0042] In one embodiment, assembly error levels can be categorized as follows: Minor errors (such as the deviation between predicted coordinates and location coordinates exceeding a preset deviation threshold) may not be flagged. Medium-level errors (such as incorrect color markings for electronic components) will be displayed as pop-up notifications. Serious errors (such as incorrect labeling of electronic components or incorrect orientation) will be alerted by an audible alarm.
[0043] In one embodiment, after determining the assembly error level, the prompting method can be determined based on the assembly error level. For example, color coding can be used, such as green for minor errors and red for serious errors; audible alarms can be used, emitting different frequencies of prompts for different levels; and screen pop-ups can be used to display error details and repair suggestions to ensure that operators can quickly identify the error type and processing priority.
[0044] By using a mechanism that determines the prompting method based on the assembly error level, the prompting method is matched with the severity of the error, thereby enabling operators to respond efficiently and improving the practicality of assembly inspection and user interaction experience.
[0045] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the detection model training method of this application. Specifically, it may include the following steps: Step S210: Use the detection model to extract features from the sample component assembly image to obtain the sample component image features corresponding to each sample electronic component, and extract features from the sample component specification text to obtain the sample component text features corresponding to each sample electronic component. The sample component assembly image includes at least two sample electronic components.
[0046] Step S220: Use the detection model to align the coordinates of the sample electronic components in the component image features and component text features to obtain the first sample alignment result.
[0047] Step S230: Align the sample electronic components in the sample component image features and sample component text features to obtain the second sample alignment sub-result.
[0048] In one implementation scenario, steps S220 and S230 can be executed in a specific order, for example, step S220 can be executed first, followed by step S230; or step S230 can be executed first, followed by step S220. In another implementation scenario, steps S220 and S230 can also be executed simultaneously, depending on the actual application, and are not limited here.
[0049] Step S240: Based on the deviation between the first sample alignment result and the first labeled data, and the deviation between the second sample alignment result and the second labeled data, adjust the network parameters of the detection model until the detection model converges.
[0050] In one implementation, a first loss value can be determined by the deviation between the first sample alignment result and the first labeled data. A second loss value can be determined by the deviation between the second sample alignment result and the second labeled data. The first and second loss values are then weighted and fused to obtain the final loss value. The network parameters of the detection model are adjusted based on the final loss value until the detection model converges. It is understood that the specific method for adjusting the network parameters of the detection model based on the deviations between the first sample alignment result and the first labeled data, and the deviations between the second sample alignment result and the second labeled data, is not limited here.
[0051] Please see Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of the electronic component assembly and inspection method of this application. Specifically, it may include the following steps: Step S310: Preprocess the component assembly image and the component specification text to be input respectively to obtain the component assembly image and the component specification text.
[0052] In one embodiment, the input component assembly image can be processed by illumination enhancement, noise removal, background segmentation, etc., to obtain a preprocessed component assembly image. Similarly, the input component specification text can be processed by removing invalid characters, standardizing terminology and units, etc., to obtain a preprocessed component specification text.
[0053] Step S320: Use the detection model to extract features from the component assembly image to obtain component image features, and extract features from the component specification text to obtain the component text features corresponding to each electronic component. The component assembly image includes multiple electronic components.
[0054] This step is the same as step S110 above, and will not be repeated here.
[0055] Step S330: Use the detection model to perform feature alignment on the component image features and component text features to obtain the alignment result corresponding to the electronic component.
[0056] This step is the same as step S120 above, and will not be repeated here.
[0057] Step S340: Analyze the alignment results to determine the assembly error level of electronic components in the component assembly image.
[0058] This step is the same as step S130 above, and will not be repeated here.
[0059] This application employs multimodal features, enabling the fusion of visual and textual features. It enhances the fine-grained classification capability of attributes by modifying the CLIP network, resulting in stronger non-linear representation of features. The detection model is trained using an open dataset, demonstrating strong category scalability and versatility.
[0060] This application also includes a real-time error handling mechanism, which can continuously enhance and expand the feature library, making identification more reliable.
[0061] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0062] Please see Figure 4 , Figure 4This is a schematic diagram of an embodiment of the electronic component assembly inspection apparatus 400 of this application. The electronic component assembly inspection apparatus 400 includes a feature extraction module 410, a feature alignment module 420, and an analysis module 430. The feature extraction module 410 performs feature extraction on the component assembly image using a detection model to obtain component image features, and performs feature extraction on the component specification text to obtain component text features corresponding to each electronic component, wherein the component assembly image includes multiple electronic components. The feature alignment module 420 performs feature alignment on the component image features and component text features using a detection model to obtain alignment results corresponding to the electronic components. The analysis module 430 performs analysis based on the alignment results to determine the assembly error level of the electronic components in the component assembly image.
[0063] In one embodiment, the feature alignment module 420 performs alignment results including a first alignment sub-result and a second alignment sub-result. It uses a detection model to perform feature alignment on component image features and component text features to obtain the alignment result corresponding to the electronic component. This includes: performing coordinate alignment on the target electronic component in the component assembly image based on the component image features and component text features to obtain the first alignment sub-result; and performing category alignment on the target electronic component based on the component image features and component text features to obtain the second alignment sub-result.
[0064] In one embodiment, the feature alignment module 420 performs coordinate alignment of a target electronic component in a component assembly image based on component image features and component text features to obtain a first alignment result, including: predicting the target electronic component based on component text features to obtain predicted coordinates, and performing target detection on component image features to obtain the position coordinates of the target electronic component; and obtaining the first alignment result based on the deviation between the predicted coordinates and the position coordinates.
[0065] In one embodiment, the feature alignment module 420 performs a first alignment result based on the deviation between the predicted coordinates and the position coordinates, including: in response to the deviation between the predicted coordinates and the position coordinates being less than or equal to a preset deviation threshold, the first alignment result is an alignment success; or, in response to the deviation between the predicted coordinates and the position coordinates being greater than the preset deviation threshold, the first alignment result is an alignment failure.
[0066] In one embodiment, the feature alignment module 420 performs category alignment of the target electronic component based on component image features and component text features to obtain a second alignment result, including: performing target detection on the component image features to determine the component category of the target electronic component in the component assembly image; filtering the component text features based on the component category to obtain several target component text features; and determining the second alignment result based on the similarity between the component image features and the target component text features.
[0067] In one embodiment, the feature extraction module 410 performs feature extraction on the component specification text, which includes the component list and process specification document corresponding to the component assembly image, to obtain the component text features corresponding to each electronic component. This includes: performing feature extraction based on the component list and process specification document to obtain the component text features corresponding to each electronic component; and after analyzing the alignment results to determine the assembly error level of the electronic components in the component assembly image, this includes: determining the prompting method based on the assembly error level.
[0068] In one embodiment, the feature alignment module 420 performs feature alignment on the component image features and component text features using the detection model to obtain the alignment result corresponding to the electronic component, and then performs the following steps: analyzes the alignment result to obtain an analysis result; in response to the analysis result being an alignment anomaly, regenerates the standard text data for the components with alignment anomalies in the component assembly image, and retrains the detection model using the electronic components with alignment anomalies and the corresponding standard text data.
[0069] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device 50 of this application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above embodiments of the electronic component assembly and detection method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.
[0070] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described embodiments of the electronic component assembly and detection method. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.
[0071] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 60 of this application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor. The program instructions 601 are used to implement the steps in any of the above embodiments of the electronic component assembly and testing method.
[0072] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0073] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0074] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A method for assembling and inspecting electronic components, characterized in that, include: The component assembly image is used to extract features by using a detection model to obtain component image features, and the component specification text is used to extract features to obtain component text features corresponding to each electronic component. The component assembly image includes multiple electronic components. The detection model is used to align the image features and text features of the component to obtain the alignment result corresponding to the electronic component. Based on the alignment results, the assembly error level of the electronic components in the component assembly image is determined.
2. The method according to claim 1, characterized in that, The alignment result includes a first alignment sub-result and a second alignment sub-result. The step of using the detection model to align the component image features and the component text features to obtain the alignment result corresponding to the electronic component includes: Based on the component image features and the component text features, coordinate alignment is performed on the target electronic components in the component assembly image to obtain the first alignment sub-result; and... Based on the component image features and the component text features, the target electronic component is categorized and aligned to obtain the second alignment sub-result.
3. The method according to claim 2, characterized in that, The step of aligning the target electronic component in the component assembly image based on the component image features and the component text features to obtain the first alignment sub-result includes: Based on the text features of the component, the predicted coordinates of the target electronic component are obtained; and based on the image features of the component, the position coordinates of the target electronic component are obtained. The first alignment result is obtained based on the deviation between the predicted coordinates and the position coordinates.
4. The method according to claim 3, characterized in that, The process of obtaining the first alignment result based on the deviation between the predicted coordinates and the position coordinates includes: In response to the deviation between the predicted coordinates and the position coordinates being less than or equal to a preset deviation threshold, the first alignment result is considered successful; or... If the deviation between the predicted coordinates and the position coordinates is greater than a preset deviation threshold, the first alignment result is an alignment failure.
5. The method according to claim 2, characterized in that, The step of aligning the target electronic component by category based on the component image features and the component text features to obtain the second alignment sub-result includes: Target detection is performed on the image features of the components to determine the component category of the target electronic components in the component assembly image; Based on the component category, the text features of the components are filtered to obtain several target component text features; The second alignment result is determined based on the similarity between the image features of the component and the text features of the target component.
6. The method according to claim 1, characterized in that, The component specification text includes a component list and process specification document corresponding to the component assembly image. The feature extraction of the component specification text to obtain the component text features corresponding to each electronic component includes: Based on the component list and process specification document, feature extraction is performed to obtain the component text features corresponding to each electronic component; After analyzing the alignment results to determine the assembly error level of the electronic components in the component assembly image, the process includes: The prompting method is determined based on the assembly error level.
7. The method according to claim 1, characterized in that, After performing feature alignment on the component image features and the component text features using the detection model to obtain the alignment result corresponding to the electronic component, the process includes: The alignment results are analyzed to obtain the analysis results; In response to the analysis result indicating an alignment anomaly, the standardized text data of the misaligned components in the component assembly image is regenerated, and the detection model is retrained using the misaligned electronic components and the corresponding standardized text data.
8. The method according to claim 1, characterized in that, The training method for the detection model is as follows: The detection model is used to extract features from the sample component assembly image to obtain the sample component image features corresponding to each sample electronic component, and to extract features from the sample component specification text to obtain the sample component text features corresponding to each sample electronic component. The sample component assembly image includes at least two sample electronic components. The detection model is used to align the coordinates of the sample electronic components in the component image features and the component text features to obtain a first sample alignment sub-result; and, The sample electronic components in the image features and text features of the sample components are aligned by category to obtain a second sample alignment sub-result; Based on the deviation between the first sample alignment result and the first labeled data, and the deviation between the second sample alignment result and the second labeled data, the network parameters of the detection model are adjusted until the detection model converges.
9. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the electronic component assembly inspection method according to any one of claims 1 to 8.
10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the electronic component assembly and inspection method according to any one of claims 1 to 8.