Vision detection guided electronic product assembly mistake proofing method and system

Abstract

The present application relates to the technical fields of machine vision and industrial automation detection, and discloses a kind of visual detection guided electronic product assembly mistake proofing method and system, comprising: obtaining product process design information and product template map;In product template map, the region of each component is framed and calibrated to generate a detection task set containing all detection targets, and template coordinates and component name are assigned;The detection task set is arranged as a guide file arranged in time sequence;Real-time acquisition of the current image of the product to be assembled, and the recognition result containing the actual detection frame coordinates of the currently installed component and the component name is identified;Sequential compliance determination is performed on the currently installed component;Image registration and position deviation calculation are performed on the actual detection frame coordinates of the currently installed component and the corresponding template coordinates in the product template map;And output the assembly status information of the currently installed component.The present application effectively avoids the decline in production efficiency.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and industrial automation inspection technology, specifically to a visual inspection-guided method and system for error prevention in electronic product assembly, applicable to quality control of manual assembly processes for multiple varieties and small batches of electronic products during the development and small-batch trial production stages of electronic equipment. Background Technology

[0002] During the research and development and small-batch trial production stages of electronic equipment, products typically exhibit characteristics of multiple varieties and small batches. The component layout and assembly requirements among similar products are highly similar, and assembly operations remain primarily manual. Although the process specifications stipulate the installation positions, models, and recommended installation sequences for various components, in actual operation, operators are prone to skipping steps based on experience and memory, omitting components at certain workstations, or installing components that look similar but have different functions. These omissions, incorrect installations, or sequential violations are often only discovered during the overall machine debugging stage, by which time irreversible potting, sealing, or high-density interconnection processes have already been initiated. Disassembly and reassembly are costly and can even lead to product scrap, making it difficult to meet high-reliability quality control requirements.

[0003] The existing quality control for manual assembly mainly adopts the following methods: First, relying on traditional process cards or work instructions for operators to check item by item; second, after the product assembly or welding is completed, using automated optical inspection (AOI), X-ray or machine vision-based defect detection systems for final inspection.

[0004] Domestic invention patent CN116228741 discloses a PCBA component defect detection method and device. It acquires PCBA images and uses a deep learning target detection model to identify the location and category of components. The detection results are compared with a standard BOM or template image to detect defects such as missing components, misaligned components, and incorrect components. Domestic invention patent CN109918523 discloses a circuit board component detection method based on the YOLO9000 algorithm. It uses YOLO series models to detect and classify components in circuit board images, and determines whether there are missing components or incorrect models by comparing with a standard component library. Domestic invention patent CN118139294 discloses a PCBA intelligent quality inspection and process optimization system. It acquires production line images to judge the quality of components and solder joints, and feeds the detection results back to optimize process parameters.

[0005] The above-mentioned technical solutions all use finished or offline static images as the processing object, which cannot make real-time judgment on the compliance of the component installation sequence during the assembly operation. It is difficult to detect operational errors such as skipping steps or violation of the sequence in a timely manner, and it is also impossible to provide operators with guidance information on the components that should be installed when errors occur.

[0006] Furthermore, Chinese invention patent CN106530293A discloses a method and system for preventing errors in manual assembly using visual inspection. This method associates assembly process guidance information and assembly status identification information according to the operation sequence, using the process number as an index to generate an error prevention early warning information model, enabling real-time monitoring and error prevention judgment during manual assembly. While this solution achieves sequential control during the assembly process, its guidance method is a strictly linear process progression. Each process number corresponds to an independent assembly action or the installation of a single part, with rigid sequential constraints between processes. Multiple parts within the same process still need to be executed one by one in a fixed order, which cannot adapt to the flexible operation requirements of installing multiple components in the same step without a specific order in actual assembly scenarios. When there is no assembly dependency between multiple components in the same product, forcing them to be installed in a fixed order may increase the operator's workload and reduce assembly efficiency. Moreover, in scenarios with frequent model changes in multi-variety, small-batch products, each model change requires redefining the complete process sequence and updating the corresponding identification template, resulting in high system deployment and maintenance costs.

[0007] Therefore, how to achieve flexible control and real-time error prevention of component installation sequence during manual assembly, while reducing system adjustment costs during product changeover, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the shortcomings of existing technologies, such as the lack of flexible sequence control in manual assembly processes and high system adjustment costs during product changeovers, this invention provides a visual inspection-guided method and system for error prevention in electronic product assembly. By selecting and calibrating on a product template drawing to generate a set of inspection tasks and arranging them into inspection steps, multiple inspection targets within the same inspection step are not subject to time sequence requirements, thus balancing assembly sequence constraints with operational flexibility. At the same time, the reuse of template drawings reduces system adjustment costs during product changeovers.

[0009] This invention is achieved through the following technical solution:

[0010] A visual inspection-guided error-proofing method for electronic product assembly includes:

[0011] Obtain product process design information and product template diagrams, wherein the product process design information includes at least the installation sequence requirements of each component;

[0012] Based on the product process design information, the areas where each component is located are selected and marked on the product template diagram to generate a detection task set containing all detection targets. Each detection target is assigned a corresponding template coordinate and component name. The detection target is the component installation position selected and marked on the product template diagram, which represents the component that should be installed at that position.

[0013] Based on the installation sequence requirements, the detection task set is arranged into a boot file arranged in chronological order. The boot file contains N ordered detection steps, where N≥1. Each detection step contains one or more detection targets, and there is no time sequence requirement between multiple detection targets within the same detection step.

[0014] The system acquires the current image of the product to be assembled in real time, performs target detection on the current image, and identifies the actual detection frame coordinates and component names containing the currently installed components.

[0015] Based on the identification results and the current expected steps, the current installed components are judged for sequential compliance, and a missing component prompt message is generated based on the result of the sequential compliance judgment.

[0016] Image registration and positional deviation calculation are performed between the actual detection frame coordinates of the currently installed components and the corresponding template coordinates in the product template diagram;

[0017] Based on the results of image registration and position deviation calculation, and combined with a preset deviation determination strategy, the assembly status information of the currently installed components is output.

[0018] Record and accumulate the assembly status information of each component until the preset completion conditions are met, and generate an assembly log.

[0019] As an optimization, based on the identification results and the current desired steps, the sequential compliance of the currently installed components is determined, including:

[0020] Based on the name of the currently installed components, query the boot file to determine the detection step to which the component belongs;

[0021] If the detection step to which the component belongs is the current expected step, then the determination order is normal;

[0022] If the detection step to which the component belongs is later than the current expected step, it is determined that there is a missing component in the current expected step, and a missing component prompt message is generated and output.

[0023] As an optimization, the detection steps for determining the component's affiliation include:

[0024] When the name of a component corresponds to only one detection step in the boot file, the component shall be assigned to that detection step.

[0025] When the component name corresponds to multiple detection steps in the guidance file, the image overlap between the actual detection box coordinates of the component and the corresponding template coordinates in the multiple detection steps is calculated respectively, and the component is assigned to the detection step with the largest image overlap.

[0026] As an optimization, the current desired step is dynamically maintained in the following manner:

[0027] Each detection step in the guidance file includes a process status field and a final status field. The process status and final status of each detection target are dynamically updated as the detection process progresses.

[0028] When the final state of all detected targets in a certain detection step is OK, the current expected step is updated to the next detection step of that detection step.

[0029] If the final state of the target being detected is not OK within a certain detection step, the detection step is maintained as the current expected step.

[0030] As an optimization, the preset deviation determination strategy includes:

[0031] Set a normal threshold and a minimum threshold for each detection target;

[0032] If the result of the image registration and position deviation calculation is greater than the normal threshold, the position is determined to be qualified.

[0033] If the result is between the normal threshold and the minimum threshold, it is determined to be a positional deviation;

[0034] If the result is less than the minimum threshold, it is determined that there is a missing item or interference.

[0035] When the judgment result is a positional deviation, missing part, or interference, a manual calibration prompt is output. In response to the external input manual confirmation command, the final status of the detected target is marked as OK, and the manual confirmation information is recorded. In response to the external input stop command, the assembly process is paused.

[0036] As an optimization, the specific process of performing target detection on the current image is as follows:

[0037] The current image is input into a preset hand detection model to determine whether there is hand occlusion; if there is, an occlusion prompt is output and the next frame of the current image is collected until no hand occlusion is detected, at which point the component recognition is performed.

[0038] Input the current image without hand occlusion into the preset segmentation model, and output the name of the installed components and the coordinates of at least one corresponding candidate detection box;

[0039] Non-maximum suppression is performed on each of the candidate detection boxes, and the candidate detection box with the highest confidence is retained as the coordinates of the actual detection box.

[0040] As an optimization, image registration and positional deviation calculation are performed on the actual detection frame coordinates of the currently installed components and the corresponding template coordinates in the product template image, including:

[0041] Feature point matching is performed between the product template image and the current image to calculate the homography matrix of the current image relative to the product template image;

[0042] Based on the homography matrix, the coordinates of each template in the product template image are mapped to the coordinate system of the current image to obtain the mapped coordinates of each detection target;

[0043] The intersection-over-union ratio (IoU) between the actual detection box coordinates and the mapped coordinates is calculated and used as the quantization result of the position deviation calculation.

[0044] As an optimization, the preset completion condition is:

[0045] All detection targets included in all detection steps within the aforementioned guidance document have accumulated assembly status information that indicates they are qualified; or,

[0046] Receive an active exit command from an external input and save the assembly status information of the currently completed inspection steps and the corresponding inspected images;

[0047] When a testing task for the same product is restarted, the saved assembly status information is automatically loaded, and execution continues from the subsequent steps of the currently completed testing steps.

[0048] As an optimization, a foreign object detection step is also included:

[0049] After acquiring the current image of the product to be assembled in real time, in response to the externally input foreign object detection command, the current image is input into a preset classification model;

[0050] The preset classification model is used to perform global foreign object identification on the current image, and the foreign object category and location coordinates are output.

[0051] The foreign object includes at least one or more of the following: gasket, wire end, thread, solder bead, screw, and masking tape.

[0052] This invention also discloses a visual inspection-guided error-proofing system for electronic product assembly, comprising:

[0053] The image acquisition unit is used to acquire real-time images of the products to be assembled.

[0054] The storage unit stores product process design information, product template diagrams, and boot files;

[0055] The processing unit executes a vision-guided electronic product assembly error prevention method as described above, and outputs the assembly status information of each component.

[0056] The prompting unit is used to output visual or audible prompts to the assembly personnel regarding the assembly status information;

[0057] The log recording unit is used to record and accumulate the assembly status information of each component, the corresponding current image and timestamp, and generate an assembly log.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] This invention, during the process preparation stage, defines the entire assembly task as an ordered sequence from Step 1 to Step n through the data structure design of the guiding file, thus constraining the sequential relationships between critical processes. Simultaneously, each Step allows for multiple inspection targets, which share the same sequence number in the data structure and are not subject to any sequential constraints. During online inspection, the system only advances to the next step after all targets within a Step have been correctly installed. This design ensures the irreversible sequence requirements of critical assembly processes while providing operational freedom for the installation of uncoupled components within the same step, effectively avoiding the production efficiency decline caused by the conflict between the forced sequence and worker operating habits in traditional strictly linear sequential guidance methods.

[0060] This invention distinguishes the cross-comparison ratio (CCR) calculation result into three levels by setting two independent judgment criteria: a normal threshold and a minimum threshold. A result greater than the normal threshold indicates accurate positioning; a result between the normal and minimum thresholds indicates a positional deviation but the component is correct; and a result less than the minimum threshold indicates that no component is installed at that position, i.e., a missing component. This dual-threshold, three-level judgment mechanism can differentiate between two different abnormal states: "the component model is correct but the installation position is incorrect" and "the component is severely misaligned or there is interference." Compared with existing technologies that only output a binary "correct / incorrect" judgment result, this avoids unnecessary rework caused by misjudging slightly misaligned components as incorrectly installed, and the quality risk of misjudging severely misaligned components as correct, leading to subsequent irreversible processes.

[0061] This invention employs a hand detection model to perform pre-occlusion detection on real-time product photos, triggering subsequent target detection tasks only when there is no hand obstruction, thus preventing missed detections and false alarms caused by operator hand occlusion. Simultaneously, an independent classification model performs global foreign object detection on the entire image, screening for unwanted objects left on the product surface due to improper handling at any time. The hand detection model, component segmentation model, and foreign object classification model work collaboratively to construct a multi-layered detection chain of "occlusion removal → component identification → foreign object screening," effectively reducing the false alarm rate caused by temporary hand occlusion, desktop clutter, and other factors.

[0062] This invention categorizes and alerts various installation errors, such as missing parts, incorrect assembly, positional deviation, and skipped steps in installation, and records assembly process images to form a traceable assembly log containing alert information, process diagrams, timestamps, and operator accounts. This provides key data support for quality control and subsequent process optimization of irreversible assembly processes in electronic products.

[0063] This invention achieves image alignment under different poses through homography matrix mapping and realizes three-level classification of position deviation through dual thresholds, further improving detection accuracy and practicality. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0065] Figure 1 This is a flowchart of a visual inspection-guided error-proofing method for electronic product assembly according to the present invention;

[0066] Figure 2 This is a detailed process diagram of a vision-guided error-proofing method for electronic product assembly according to the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0068] Example 1 discloses a visual inspection-guided method for error prevention in electronic product assembly, such as... Figure 1-2 As shown, it includes 8 steps, which will now be explained in detail.

[0069] S1. Obtain product process design information and product template diagram. The product process design information shall at least include the installation sequence requirements of each component.

[0070] Obtain product process design information and product template drawings. Product process design information includes product drawing number (unique string) and process step set (containing sequential process step codes, component names, and component quantities). Product template drawings are physical product images or models with 3D rendering effects, captured by photographing the product horizontally placed within the field of view of a fixed industrial camera, with the product edges parallel to the edge of the camera's field of view and a rotation angle not exceeding ±5° to ensure standardization of the template drawings.

[0071] S2. Based on the product process design information, the areas where each component is located are selected and marked on the product template drawing to generate a detection task set containing all detection targets. The corresponding template coordinates and component names are assigned to each detection target. The detection target is the component installation position selected and marked on the product template drawing, which represents the component that should be installed at that position.

[0072] Based on the product process design information, process engineers select and mark the areas where each component is located on the product template drawing. Specifically, they use a mouse or other input device to draw a rectangle on the product template drawing, selecting the area where any component is located, thus forming the template coordinates of that component. Simultaneously, they select the component name corresponding to that location from the process steps set in the process design information, completing the definition of the inspection target. This process is repeated until all components requiring installation on the product template drawing have been selected and marked, forming an inspection task set containing all inspection targets.

[0073] After the selection and calibration are completed, the system automatically compares the total number of inspection targets in the inspection task with the number of components in the product process design information. If the two are inconsistent, a message will be displayed indicating a missing item or an excess. Process personnel will then add missing components or delete redundant inspection targets according to the prompts until the numbers match, ensuring the completeness of the inspection targets.

[0074] S3. Based on the installation sequence requirements, the detection task set is arranged into a boot file arranged in chronological order. The boot file contains N ordered detection steps, where N≥1. Each detection step contains one or more detection targets, and there is no time sequence requirement between multiple detection targets within the same detection step.

[0075] Based on the installation sequence requirements in the product process design information, the test task set is arranged into a chronologically ordered guide file. The guide file contains N ordered test steps (Step 1 to Step n), where N is an integer greater than or equal to 1.

[0076] Each testing step contains one or more testing targets. When a testing step contains multiple testing targets, there is no requirement for the testing time order among these targets. Operators can install the components in this step in any order according to their operating habits or actual assembly situation, and the system will not trigger an alarm due to the different installation order of the targets within this step. There are time constraints between different testing steps, that is, Step 2 can only be entered after all of Step 1 is completed, Step 3 can only be entered after all of Step 2 is completed, and so on.

[0077] Each inspection step in the guidance file also includes a process status field and a final status field. The process status and final status of each inspection target are dynamically updated as the inspection process progresses, recording the real-time status and final judgment result of each inspection target during the assembly process. The process status records temporary judgment results during the inspection process (such as OK, NG, and the specific type of NG), and can be updated multiple times as the number of inspections increases; the final status is the final judgment conclusion of the inspection target, which is not changed once determined.

[0078] More specifically, the data structure for each testing step in the guidance file mainly includes fields such as sequence number, component name, guidance diagram, process status, process diagram, final status, and result diagram. The sequence number uniquely identifies the location of the testing target in the guidance file; the component name corresponds to the component that should be installed at that location in the process design information; the guidance diagram is a screenshot of the selected area corresponding to the testing target on the template diagram, used to assist operators in identifying the target location during testing. The process status and process diagram record temporary judgment results and corresponding real-time photos during the testing process, and can be updated multiple times as the number of tests increases; the final status and result diagram represent the final judgment conclusion for the testing target and the corresponding confirmation photo, and are not changed once determined. A testing task may contain and save multiple process statuses and corresponding process diagrams with inconsistent prompts, but the final status and result diagram are unique.

[0079] Taking the assembly of a certain product as an example, the data structure of the boot file is shown in Table 1:

[0080] Table 1. Data Structure of Boot Files

[0081] Step Serial Number Component Name Guide Map Process status Process diagram Final state Result image Step 1 1-1 A Already bound OK Saved OK Saved Step 1 1-2 B Already bound NG (Component Mismatch) Saved OK Saved Step 2 2-1 C Already bound NG (Position Deviation) Saved OK Saved Step 2 2-2 A Already bound NG (Missing item) Saved OK Saved Step 3 3-1 A Already bound OK Saved OK Saved Step 3 3-2 D Already bound NG (Missing item) Saved OK Saved Step 3 3-3 E Already bound Recognition in progress Retrieving — —

[0082] As shown in Table 1, Step 2 includes two detection targets, C and A, with no requirement for a specific detection time order. The process and final states of each detection target are recorded independently. In the guidance diagram, "Bound" indicates that the detection target has been associated with its corresponding template coordinate region.

[0083] S4. Real-time acquisition of the current image of the product to be assembled, and target detection of the current image to identify the actual detection frame coordinates and component names containing the currently installed components.

[0084] After the assembly personnel log into the system, the system uses an image acquisition unit (such as an industrial camera fixedly installed above the assembly station) to acquire real-time images of the products to be assembled.

[0085] The target detection process for the current image includes three sub-steps: hand occlusion detection, component identification, and NMS deduplication.

[0086] First, the current image is input into a pre-set hand detection model to determine if any hand occlusion exists in the image. The preferred pre-set hand detection model uses the MediaPipe hand detection algorithm, which can output the presence and location of hand features in the image. If hand occlusion is detected, an occlusion prompt is output (such as a visual or audible prompt displaying "Please remove your hands"), and the system continues to acquire the next frame of the current image until no hand occlusion is detected, at which point component recognition continues. This pre-detection step prevents missed detections and false alarms caused by operator hand occlusion from the outset.

[0087] Then, the current image without hand occlusion is input into a pre-set segmentation model to identify whether the current image contains a target. If no target is detected, a missing part is indicated; if a target is detected, the name of the installed component and the coordinates of at least one corresponding candidate detection box are output. The pre-set segmentation model is preferably a pre-trained UltralyticsYOLO26-seg instance segmentation model. This model is obtained by pre-constructing a database containing component images and corresponding component names and training it for a specified number of epochs. It can output the category label and detection box coordinates for each detected component.

[0088] Finally, non-maximum suppression (NMS) is performed on each candidate detection box, retaining the candidate detection box with the highest confidence as the actual detection box coordinates. Specifically, the intersection-over-union (IoU) ratio of the baseline box to other boxes is calculated. All candidate detection boxes are sorted from highest to lowest confidence, and the box with the highest confidence is selected as the baseline box and retained. Boxes with an IoU ratio exceeding a preset NMS suppression threshold are suppressed and deleted. The above process is repeated for the remaining boxes until all boxes have been processed. The NMS suppression threshold is preferably set to 0.8. Through NMS processing, it is ensured that each component ultimately corresponds to only one detection box coordinate, avoiding interference from duplicate detection boxes in subsequent IoU calculations.

[0089] like Figure 2As shown, the top image is the original detection result, and the bottom image is the detection result after NMS processing. In the original detection result above, multiple candidate detection boxes exist around the same component, each with a different confidence score. After NMS processing on the right, only the detection box with the highest confidence score is retained for the same component, and duplicate candidate boxes are suppressed and deleted. By comparison, it can be seen that NMS processing can effectively eliminate duplicate detection of the same target, ensuring that each component ultimately corresponds to only one detection box coordinate, providing accurate input data for subsequent sequence determination and positional deviation calculation.

[0090] It should be noted that, in any process of step S4, in response to the externally input foreign object detection command, the current image is input into the preset classification model, and the current image is globally identified by the preset classification model, and the foreign object category and foreign object location coordinates are output.

[0091] Foreign object detection does not require a template image; its processing flow is a separate target detection process. A pre-built classification model directly performs global target detection on the input current image. When a foreign object is detected, it outputs the name and coordinates of the foreign object and visually labels it on the image. The foreign object category is defined before training the pre-built classification model, based on the types of foreign objects commonly encountered in the environment. In this embodiment, the foreign object category includes one or more of the following: washers, wire ends, threads, solder beads, screws, and masking tape. The specific category can be added or removed based on the environmental characteristics of the actual application scenario.

[0092] The pre-set classification model is preferably a pre-trained YOLO11 model, obtained by pre-constructing a database containing foreign object images and corresponding foreign object names and training it for a specified number of rounds. Foreign object detection and component identification are independent of each other and can be triggered at any time during the assembly inspection process without affecting the main inspection flow.

[0093] The system has two operating modes: active and automatic. In active mode, the operator manually clicks the photo button after installing each component, and the system captures the current image and performs target detection. In automatic mode, the system automatically captures an image at fixed time intervals (e.g., 2 seconds) and performs target detection. In automatic mode, when the system detects an NG (Not Valid) state, the operator must manually click "OK" or "Manual Approval." Clicking either option saves the photo and status; the difference is that clicking "Manual Approval" directly saves the final status as "OK."

[0094] S5. Based on the identification result and the current expected steps, determine the order compliance of the currently installed components, and determine whether to generate a missing component prompt message based on the result of the order compliance determination.

[0095] Based on the identification results (component name and actual detection box coordinates) obtained in step S4 and the current expected steps, the sequential compliance of the currently installed components is determined.

[0096] First, based on the name of the currently installed components, query the boot file to determine the testing step to which the component belongs. The specific determination method is as follows:

[0097] When the component name corresponds to only one detection step in the boot file, the component is directly assigned to that detection step.

[0098] When a component name corresponds to multiple detection steps in the guidance file (i.e., the same component appears repeatedly in different steps), the image overlap ratio (i.e., intersection-union ratio) between the actual detection box coordinates of the component and the corresponding template coordinates in each of the multiple detection steps is calculated, and the component is assigned to the detection step with the highest intersection-union ratio. This disambiguation mechanism distinguishes the assignment of the same component name in different detection steps through spatial location information, solving the problem of determining the recurrence of the same component name in multiple workstations.

[0099] After determining the detection step to which the component belongs, the judgment is made based on its sequence relationship with the currently desired step:

[0100] If the detection step to which the component belongs is the current expected step, then the judgment sequence is normal, and proceed to step S6 for position deviation calculation.

[0101] If the detection step to which the component belongs is later than the current expected step, it is determined that there is a missing component in the current expected step, and a missing component prompt message is generated and output. The missing component prompt message includes the string "missing component" and the corresponding component name and quantity. For example, if the current expected step is Step 2, but the system detects a component at Step 3, it indicates that one or more components in Step 2 have not been installed, and the system determines that Step 2 is missing and outputs a prompt message.

[0102] The current expected steps are dynamically maintained in the following manner:

[0103] Each detection step in the guidance file includes a process status field and a final status field. The process status and final status of each detection target are dynamically updated as the detection process progresses. When the final status of all detection targets within a certain detection step is OK, the current desired step is updated to the next detection step of that step; when the final status of any detection target within a certain detection step is not OK, that detection step remains unchanged as the current desired step. This "state convergence-driven advancement" mechanism, unlike the traditional workflow that relies on user click confirmation, achieves automated management of assembly progress.

[0104] The current expected step, also known as the "first priority" in the boot file, is dynamically maintained as follows: when the final state of all detected targets within a certain detection step is OK, the detection step is considered complete, and the first priority is updated to the next detection step of that detection step; when the final state of a detected target within a certain detection step is not OK, the first priority remains unchanged and still points to that detection step.

[0105] Taking Table 1 as an example, assume the first priority is Step 2, which includes 2-1 (C) and 2-2 (A). After the system identifies component "A", it searches the entire boot file and finds that "A" appears in three places: Step 1 (1-1), Step 2 (2-2), and Step 3 (3-1), which is not unique. The system calculates the intersection-union ratio (IUU) of the detection box coordinates of the currently identified A with the coordinates of the three template mappings. The system finds that the sequence number corresponding to the maximum IUU is 3-1 in Step 3. Therefore, it determines that A belongs to Step 3 and does not belong to the first priority. Since the attribution step (Step 3) is later than the first priority (Step 2), the system checks the final state of Step 2 and finds that the final state of A in 2-2 is not yet OK, and the position is empty (no component is identified). Therefore, it determines that 2-2 in Step 2 is missing, generates and outputs a missing component prompt message, and the first priority remains unchanged in Step 2 until the final state of all targets in Step 2 is OK before proceeding to Step 3.

[0106] Conversely, if the currently identified A, after disambiguation, belongs to 2-2 in Step 2, and the first priority is Step 2, then the sequence is normal, and proceed to S6 to calculate the position deviation.

[0107] A detection task can include and save multiple process states with inconsistent prompts and corresponding process diagrams, but the final state and result diagram are unique. As shown in Table 1, if the worker does not process 2-2 in the end (no state change occurs), the most recent process state and process diagram will be saved as the final state and result diagram when the log is generated.

[0108] Through the aforementioned dynamic maintenance mechanism of "first priority", the system can accurately identify skipped installation behavior. When the system identifies a component that does not belong to the first priority, it can infer that there is a missing component in the detection step corresponding to the first priority, thereby achieving real-time error prevention of assembly sequence.

[0109] This solution defines the assembly task as an ordered sequence from Step 1 to Step n, where each Step contains one or more inspection targets, and there is no time order requirement between multiple inspection targets within the same Step. Through dynamic maintenance of the "Process Status" and "Final Status" fields, the first target in a Step only proceeds to the next step when the final status of all targets within the Step is OK. This allows the invention to ensure irreversible sequential constraints on critical processes while providing operational freedom for uncoupled devices within the same step, avoiding increased waiting time and decreased assembly efficiency caused by conflicts between forced sequence and worker operating habits. This effect is not disclosed in existing technical literature.

[0110] Furthermore, this invention employs a "reverse tracing" logic: when the system identifies a component belonging to the next step, it backtracks to check the final state of the previous step; if an unfinished target exists, it is determined to be missing. The coordinates of the detection box are determined by IoU disambiguation, and a missing component warning is triggered if the assigned step is later than the currently expected step. This allows for the discovery of missing components in subsequent steps, and the determination is made through the timing misalignment information of "belonging to the next step but the previous step not being completed." This eliminates the need for real-time verification of each process to detect missing components, reducing the system burden of having to independently set detection points for each process.

[0111] S6. Perform image registration and position deviation calculation on the actual detection frame coordinates of the currently installed components and the corresponding template coordinates in the product template diagram.

[0112] Image registration and positional deviation calculation are performed on the actual detection frame coordinates obtained after NMS deduplication in step S4 and the corresponding template coordinates in the product template image. Specifically, this includes:

[0113] Feature point matching is performed between the product template image and the current image to calculate the homography matrix of the current image relative to the product template image. Specifically, feature points are extracted from both the product template image and the current image (using feature extraction algorithms such as SIFT and ORB). Point pairs are established through feature matching, and robust estimation is performed using the RANSAC (Random Sample Consensus) algorithm (e.g., calling the cv2.RANSAC function) to solve for the homography matrix H between the two images. The RANSAC algorithm, through iterative random sampling, can stably estimate the optimal homography matrix even in the presence of mismatched points.

[0114] The homography matrix maps the coordinates of each template in the product template image to the coordinate system of the current image, thus obtaining the mapped coordinates of each detected target. Specifically, for any point P (homogeneous coordinates) on the template image, its corresponding point P' mapped to the current image coordinate system is calculated using the formula P' = H·P. Through homography matrix mapping, even if the position and pose of the product in the current image differ from those in the template image (such as translation or slight rotation of the product within the field of view), the template coordinates can be accurately projected to the corresponding position in the current image, achieving precise alignment under different poses.

[0115] The Intersection over Union (IoU) ratio between the actual detection box coordinates and the mapped coordinates is calculated as a quantification result of the positional deviation calculation. The formula for calculating the IoU ratio is: IoU = |A∩B| / |A∪B|, where A is the actual detection box area and B is the theoretical installation area corresponding to the template mapped coordinates. The IoU value range is [0,1]. The closer the value is to 1, the higher the degree of overlap between the two boxes and the smaller the positional deviation; the closer the value is to 0, the larger the positional deviation.

[0116] S7. Based on the results of image registration and position deviation calculation, and combined with a preset deviation determination strategy, output the assembly status information of the currently installed components.

[0117] Based on the crossover ratio calculated in step S6, and combined with the preset deviation judgment strategy, the assembly status information of the currently installed components is output.

[0118] S7.1 Design a dual threshold decision logic.

[0119] The threshold comparison results include the following three cases:

[0120] When the cross-comparison (CUC) result exceeds the normal threshold, a further comparison is made to see if the identified name matches the component name in the detection task. If they match, the assembly is considered correct; if they do not match, the assembled components are considered incorrect.

[0121] When the cross-comparison (CUC) result is less than the normal threshold but greater than the minimum threshold, a further comparison is made to see if the identified name matches the component name in the detection task. If they match, it is determined that the assembly is correct but the position is off; if they do not match, it is determined that the assembled component is incorrect.

[0122] When the cross-comparison ratio is less than the minimum threshold, it is determined that no component is installed at that position, which is called missing component (NG). The missing component prompt is output, which includes the string "missing component" and the corresponding component name and quantity.

[0123] Positional deviations include translational deviations and rotational deviations.

[0124] S7.2 Threshold Determination and Configuration Method.

[0125] Basic method: Manual experimentation and statistical method.

[0126] During the prototype assembly phase, threshold values ​​were determined through multiple tests: Components were manually judged and marked for misalignment. For correctly installed components without misalignment, their cross-parallel ratio (CPR) was calculated, and the minimum value was taken as the normal threshold. For components with misalignment, their CPR was calculated, and the minimum value was taken as the minimum threshold. In this embodiment, the normal threshold is preferably 0.7~0.85, and the minimum threshold is preferably 0.3~0.5.

[0127] Preferred approach: Threshold configuration based on boot file grouping structure.

[0128] Based on the grouped structure of the guidance file, the types of components, local density, and accuracy requirements involved in different detection steps may vary. Therefore, this invention independently configures normal thresholds and minimum thresholds for each detection target and stores them as part of the guidance file. During detection, the system automatically calls the threshold corresponding to the detection step to which the currently installed component belongs to determine positional deviation. When the same component name appears in different detection steps, different thresholds can be configured for each step due to their different installation scenarios, without affecting each other.

[0129] In addition to the preferred method, the following methods can also be used to help determine the threshold of each detection target:

[0130] 1. Based on the IPC standard mapping baseline threshold.

[0131] According to the IPC-A-610G standard, different types of components have different offset tolerance requirements: 25%~50% of the solder tip width for chip resistors / capacitors, 25%~50% of the lead width for QFP and other wing-pin devices, and 15%~25% of the ball diameter for BGAs. The system automatically queries the corresponding IPC tolerance level based on the component type marked by the process engineer during selection and mapping, and maps it to an IoU threshold: component types with stricter tolerance requirements (such as BGAs in Level 3 products, requiring offset ≤ 15% of the ball diameter) are assigned a higher normal threshold (e.g., 0.85); component types with more lenient tolerance requirements (such as chip resistors in Level 1 products, allowing offset ≤ 50% of the solder tip width) are assigned a lower normal threshold (e.g., 0.70).

[0132] 2. Based on local density correction threshold.

[0133] In densely populated areas of PCBA components, grayscale changes are frequent, and the difference between the background and the target is small. Traditional global threshold segmentation is inaccurate. The nearest-neighbor local adaptive thresholding method can improve the segmentation effect. The system calculates the local density at the location of each detected target on the template image. Centered on the template coordinates of the current detected target, it counts the number N of adjacent detected targets within a radius R (R can be set to 10~20mm, adjusted according to product size), and calculates the local density. Density correction factor ,in This represents the maximum density value of all detected targets in the product template image. To correct for the intensity coefficient (ranged from 0.02 to 0.05), the normal threshold is appropriately lowered (by 0.03 to 0.08) for higher local density, in order to reduce false alarms caused by decreased detection box accuracy due to interference from adjacent targets. The final normal threshold is... ,in The baseline normal threshold is used; the final minimum threshold is calculated in the same way.

[0134] 3. Calibrate thresholds based on data distribution statistics.

[0135] For new products, the system collects images of 20-30 correctly installed components during the process preparation stage, calculates the IoU distribution of each detection target, and calibrates the benchmark threshold based on the statistical distribution characteristics, so that the threshold can be interpreted according to industry standards and can adapt to actual imaging conditions through experimental data.

[0136] 4. Dynamically fine-tune the threshold based on historical Step data.

[0137] When the same component name appears repeatedly in multiple steps, the system accumulates and records the actual IoU values ​​of that component in the completed steps, and calculates its historical average. When the component is identified again in a subsequent step, the system compares the current IoU value with the historical average. If the current IoU value continues to deviate from the historical average (the deviation exceeds a certain threshold), the system will terminate the process. The range indicates a potential systematic deviation (such as changes in lighting or product placement position drift). The system automatically and appropriately lowers the threshold requirement for this component in this step (e.g., lowering it by 0.03~0.05) to avoid misjudgments caused by environmental changes. Simultaneously, the system continuously adds new detection results to the historical database and dynamically updates it. This allows the threshold to adaptively adjust in response to changes in the environment.

[0138] Using the above method, the threshold is no longer a one-time calculation based on a static sample set, but rather a dynamic evolution based on historical IoU data accumulated during the detection process. Its difference from existing adaptive thresholding schemes lies in the fact that existing schemes adjust based on factors unrelated to the assembly process, while this scheme uses the temporal structure of each step in the assembly process as an independent dimension for threshold calibration, making the threshold adaptable not only to the image content but also to the assembly process state.

[0139] Output the corresponding prompt message based on the judgment result:

[0140] When the assembly is determined to be correct, output a "Correct" message;

[0141] When it is determined that the assembled components are inconsistent, the message "Component Inconsistent" will be output.

[0142] When the assembly is determined to be correct but there is a positional deviation, a "positional deviation" prompt is output, and the manual calibration process is initiated.

[0143] When it is determined that there is a missing item or interference, a "manual verification" prompt will be output, and the manual verification process will begin.

[0144] In the manual verification process, in response to an externally inputted manual confirmation command, the final status of the inspection target is marked as OK; in response to an externally inputted stop command, the assembly process is paused.

[0145] At the same time, the prompt message, the corresponding real-time product photo, the time the photo was acquired, and the account information of the currently logged-in assembly personnel are recorded as part of the assembly log.

[0146] S8. Record and accumulate the assembly status information of each component until the preset completion conditions are met, and generate an assembly log.

[0147] Repeat steps S4 to S7, and record the assembly status information of each component as it is identified, accumulating the status changes of each component from the first identification to the final confirmation.

[0148] When the preset completion conditions are met, the inspection process ends and an assembly log is generated. The preset completion conditions are: all inspection targets included in all inspection steps in the guide file have accumulated assembly status information that is qualified (i.e., the final status of all components is OK); or, receiving an active exit command from an external input.

[0149] The assembly log is a collection of all records, including prompts for each component, corresponding process diagrams / result diagrams, the time of photo acquisition, the account information of the currently logged-in assembly personnel, and the complete state change trajectory of each component from initial identification to final confirmation. A single inspection task may include and save multiple process states with inconsistent prompts and corresponding process diagrams, but the final state and result diagram are unique. If the operator does not process a certain inspection target (no state change occurs), the most recent process state and process diagram will be saved as the final state and result diagram when the log is generated.

[0150] When an assembly worker voluntarily exits the system, the system saves the assembly status information and corresponding inspected images for the currently completed inspection steps. When the same assembly worker logs into the system again and selects to continue the task, the system automatically loads the saved assembly status information and continues execution from the subsequent steps of the currently completed inspection steps, realizing the continuation of the assembly task from the breakpoint and avoiding repetitive work caused by interruption.

[0151] If the operator does not process a certain detection target (no state change occurs), the most recent process state and process diagram will be saved as the final state and result diagram when the log is generated.

[0152] In summary, compared with the prior art, the present invention is substantially different in the following aspects:

[0153] 1. In existing technologies, some solutions extract the geometric features of operators' gestures by collecting videos of manual assembly samples and use them as part of the assembly behavior recognition. Such solutions require pre-collecting a large number of gesture samples and training a gesture classifier, resulting in high system complexity. This invention treats the hand as an interference source in the detection process, employing a hand detection model for pre-occlusion detection. Subsequent component recognition and judgment are only triggered when there is no hand occlusion. This eliminates the need to train a gesture classifier, simplifying the processing flow and reducing system deployment and maintenance costs.

[0154] 2. Existing technologies use process numbers as indexes and proceed linearly, with each process number corresponding to an independent assembly action or the installation of a single part, resulting in rigid sequence constraints between processes. This approach cannot support the free installation of multiple components within the same process without a specific order, and for scenarios where multiple identical components need to be installed in a specific order, it is necessary to distinguish them one by one based on the process number, making the process arrangement relatively rigid. This invention achieves sequence constraints through a Step grouping structure in the guidance file: components with sequence constraints are grouped into different Steps, while components without sequence constraints are grouped into the same Step. Within the same Step, multiple detection targets have no time order requirements, and operators can install them freely according to their operating habits; there are time constraints between different Steps, and the system only advances the first priority to the next Step when the final state of all detection targets in the current Step is OK. At the same time, for multiple identical components, this invention uniquely identifies their position in the guidance file through a combination of "component name + template coordinates," combined with a dynamic maintenance mechanism for the first priority, allowing the system to clearly determine which component should be installed, thereby forcibly constraining the installation order of identical components.

[0155] 3. Existing technologies typically output binary results as "correct / incorrect" or "qualified / unqualified". This invention employs a dual-threshold, three-level judgment mechanism, categorizing the cross-comparison ratio calculation result into three cases: greater than the normal threshold, between the two thresholds, and less than the minimum threshold. These correspond to three judgment results: "position qualified," "position deviation," and "missing component or interference," respectively, outputting differentiated prompts such as "correct," "component mismatch," "position deviation," and "manual verification." Furthermore, this invention supports manual intervention for confirmation. For detection targets judged as having positional deviation or missing / interference issues, the system allows operators to manually mark the final status as OK and record the manual confirmation information. This enables the system to adapt to the needs of flexible manual judgment in actual assembly scenarios, avoiding production efficiency losses caused by indiscriminate line stoppages.

[0156] Existing technologies primarily output the final assembly conformity conclusion, lacking a systematic recording and traceability capability for abnormal states that occur during the assembly process. This invention independently records the process state and final state for each inspection target. The process state records the temporary judgment result and corresponding process diagram for each inspection, which can be updated multiple times as the number of inspections increases. The final state is the final judgment conclusion and corresponding result diagram for the inspection target, which remains unchanged once determined. When an inspection target is finally confirmed as OK after experiencing multiple NGs (Not Acceptable Errors), the system completely retains the complete process state change trajectory from NG to OK and the corresponding process image. This design allows for subsequent statistical analysis of the types of errors that occurred during the assembly process of each component and the frequency of each error type. This facilitates process optimization and operational training for process personnel targeting high-frequency error points, providing data support for quality traceability of irreversible assembly processes in military electronics.

[0157] In summary, the present invention differs substantially from the prior art in multiple dimensions, and has significant technical advantages in the unification of flexible control and rigid constraints of assembly sequence, the refined classification of detection and judgment, flexible human intervention, and process data traceability.

[0158] Example 2 also discloses a visual inspection-guided error-proofing system for electronic product assembly, comprising:

[0159] The image acquisition unit is used to acquire real-time images of the products to be assembled.

[0160] The storage unit stores product process design information, product template diagrams, and boot files;

[0161] The processing unit executes a visual inspection-guided electronic product assembly error prevention method as described in Example 1, and outputs the assembly status information of each component.

[0162] The prompting unit is used to output visual or audible prompts to the assembly personnel regarding the assembly status information;

[0163] The log recording unit is used to record and accumulate the assembly status information of each component, the corresponding current image and timestamp, and generate an assembly log.

[0164] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual inspection-guided method for error prevention in electronic product assembly, characterized in that, include: Obtain product process design information and product template diagrams, wherein the product process design information includes at least the installation sequence requirements of each component; Based on the product process design information, the areas where each component is located are selected and marked on the product template diagram to generate a detection task set containing all detection targets. Each detection target is assigned a corresponding template coordinate and component name. The detection target is the component installation position selected and marked on the product template diagram, which represents the component that should be installed at that position. Based on the installation sequence requirements, the detection task set is arranged into a boot file arranged in chronological order. The boot file contains N ordered detection steps, where N≥1. Each detection step contains one or more detection targets, and there is no time sequence requirement between multiple detection targets within the same detection step. The system acquires the current image of the product to be assembled in real time, performs target detection on the current image, and identifies the actual detection frame coordinates and component names containing the currently installed components. Based on the identification results and the current expected steps, the current installed components are judged for sequential compliance, and a missing component prompt message is generated based on the result of the sequential compliance judgment. Image registration and positional deviation calculation are performed between the actual detection frame coordinates of the currently installed components and the corresponding template coordinates in the product template diagram; Based on the results of image registration and position deviation calculation, and combined with a preset deviation determination strategy, the assembly status information of the currently installed components is output. Record and accumulate the assembly status information of each component until the preset completion conditions are met, and generate an assembly log.

2. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 1, characterized in that, Based on the identification results and the current expected steps, the sequential compliance of the currently installed components is determined, including: Based on the name of the currently installed components, query the boot file to determine the detection step to which the component belongs; If the detection step to which the component belongs is the current expected step, then the determination order is normal; If the detection step to which the component belongs is later than the current expected step, it is determined that there is a missing component in the current expected step, and a missing component prompt message is generated and output.

3. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 2, characterized in that, The testing steps to determine the component's classification include: When the name of a component corresponds to only one detection step in the boot file, the component shall be assigned to that detection step. When the component name corresponds to multiple detection steps in the guidance file, the image overlap between the actual detection box coordinates of the component and the corresponding template coordinates in the multiple detection steps is calculated respectively, and the component is assigned to the detection step with the largest image overlap.

4. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 2, characterized in that, The current desired step is dynamically maintained in the following manner: Each detection step in the guidance file includes a process status field and a final status field. The process status and final status of each detection target are dynamically updated as the detection process progresses. When the final state of all detected targets in a certain detection step is OK, the current expected step is updated to the next detection step of that detection step. If the final state of the target being detected is not OK within a certain detection step, the detection step is maintained as the current expected step.

5. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 1, characterized in that, The preset deviation determination strategy includes: Set a normal threshold and a minimum threshold for each detection target; If the result of the image registration and position deviation calculation is greater than the normal threshold, the position is determined to be qualified. If the result is between the normal threshold and the minimum threshold, it is determined to be a positional deviation; If the result is less than the minimum threshold, it is determined that there is a missing item or interference. When the judgment result is a positional deviation, missing part, or interference, a manual calibration prompt is output. In response to the external input manual confirmation command, the final status of the detected target is marked as OK, and the manual confirmation information is recorded. In response to the external input stop command, the assembly process is paused.

6. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 1, characterized in that, The specific process of performing target detection on the current image is as follows: The current image is input into a preset hand detection model to determine whether there is hand occlusion; if there is, an occlusion prompt is output and the next frame of the current image is collected until no hand occlusion is detected, at which point the component recognition is performed. Input the current image without hand occlusion into the preset segmentation model, and output the name of the installed components and the coordinates of at least one corresponding candidate detection box; Non-maximum suppression is performed on each of the candidate detection boxes, and the candidate detection box with the highest confidence is retained as the coordinates of the actual detection box.

7. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 1, characterized in that, Image registration and positional deviation calculation are performed on the actual detection frame coordinates of the currently installed components and the corresponding template coordinates in the product template image, including: Feature point matching is performed between the product template image and the current image to calculate the homography matrix of the current image relative to the product template image; Based on the homography matrix, the coordinates of each template in the product template image are mapped to the coordinate system of the current image to obtain the mapped coordinates of each detection target; The intersection-over-union ratio (IoU) between the actual detection box coordinates and the mapped coordinates is calculated and used as the quantization result of the position deviation calculation.

8. The method for preventing errors in electronic product assembly guided by visual inspection according to claim 1, characterized in that, The preset completion conditions are: All inspection targets included in all inspection steps within the aforementioned guidance document have accumulated assembly status information that indicates they are qualified; or, Receive an active exit command from an external input and save the assembly status information of the currently completed inspection steps and the corresponding inspected images; When a testing task for the same product is restarted, the saved assembly status information is automatically loaded, and execution continues from the subsequent steps of the currently completed testing steps.

9. A visual inspection-guided error-proofing method for electronic product assembly according to claim 1, characterized in that, It also includes a foreign object detection step: After acquiring the current image of the product to be assembled in real time, in response to the externally input foreign object detection command, the current image is input into a preset classification model; The preset classification model is used to perform global foreign object identification on the current image, and the foreign object category and location coordinates are output. The foreign object includes at least one or more of the following: gasket, wire end, thread, solder bead, screw, and masking tape.

10. A visual inspection-guided error-proofing system for electronic product assembly, characterized in that, include: The image acquisition unit is used to acquire real-time images of the products to be assembled. The storage unit stores product process design information, product template diagrams, and boot files; The processing unit executes a vision-guided electronic product assembly error prevention method as described in any one of claims 1 to 8, and outputs the assembly status information of each component. The prompting unit is used to output visual or audible prompts to the assembly personnel regarding the assembly status information; The log recording unit is used to record and accumulate the assembly status information of each component, the corresponding current image and timestamp, and generate an assembly log.

Images