Electronic component detection system and method

Abstract

The electronic component detection system comprises a multi-modal data acquisition module, an image enhancement and generation module, a multi-modal feature fusion and decision module and a result display and control module, the image enhancement and generation module is electrically connected with the multi-modal data acquisition module, and the image enhancement and generation module is electrically connected with the multi-modal data acquisition module. The multi-modal feature fusion and decision-making module is electrically connected with the image enhancement and generation module and the multi-modal data acquisition module, and the result display and control module is electrically connected with the multi-modal feature fusion and decision-making module. According to the method, the working condition of poor data quality or temporary failure of a single sensor is effectively handled, the continuity of a detection process and the integrity of decision information are ensured, and a final report not only comprises defect classification and positions, but also records a complete detection decision path and key data, so that quality tracing, model optimization and manual rechecking are facilitated.

Description

Technical Field

[0001] This invention relates to the field of electronic component technology, and in particular to an electronic component testing system and method. Background Technology

[0002] With the trend of highly integrated and miniaturized electronic products, the quality inspection of electronic components is crucial. Traditional inspection methods often rely on a single modality, such as manual visual inspection or a single type of automated optical inspection, which suffers from low efficiency, strong subjectivity, and insufficient ability to detect internal and electrical defects. Although multimodal inspection systems employing multiple sensors have emerged in existing technologies, the following problems still exist: 1. Uneven data quality or occasional missing data from different modalities affects the overall reliability of judgment; 2. The multimodal information fusion method is simple and fails to deeply explore the complex correlations and complementarities between modalities; 3. The inspection process is fixed and rigid, unable to dynamically adjust the strategy based on the currently detected clues, resulting in low efficiency when facing complex or atypical defects, or the overuse of high-cost inspection methods in pursuit of reliability. Therefore, there is an urgent need for an electronic component inspection system and method that can intelligently handle data incompleteness, achieve deep feature fusion, and adaptively optimize the inspection process. Summary of the Invention

[0003] The present invention aims to at least partially solve one of the technical problems in the related art.

[0004] Therefore, the purpose of this invention is to provide an electronic component testing system and method.

[0005] To achieve the above objectives, this invention proposes an electronic component inspection system and method, comprising a multimodal data acquisition module, an image enhancement and generation module, a multimodal feature fusion and decision module, and a result display and control module. The multimodal data acquisition module is used to synchronously or asynchronously acquire at least two different types of data from the electronic component to be inspected. The data types include optical image data, electrical signal waveform data, and internal structure imaging data. The image enhancement and generation module, electrically connected to the multimodal data acquisition module, is used to assess the quality of the acquired image data and enhance or generate images with poor quality or missing data based on other modal data. The multimodal feature fusion and decision module, electrically connected to both the image enhancement and generation module and the multimodal data acquisition module, is used to extract and fuse features from different modal data, perform defect detection and classification based on the fused features, and dynamically adjust the detection strategy. The result display and control module, electrically connected to the multimodal feature fusion and decision module, is used to display the inspection results, provide an interactive interface, and control the system to perform refined re-inspection actions.

[0006] In addition, the electronic component testing system and method proposed above according to the present invention may also have the following additional technical features:

[0007] Specifically, the image enhancement and generation module includes a quality assessment unit, a data enhancement unit, and a cross-modal generation unit. The quality assessment unit is used to evaluate the sharpness, contrast, or integrity of image data. The data enhancement unit is used to enhance poor-quality image data based on bilateral filtering, edge enhancement, and detail fusion algorithms. The cross-modal generation unit is used to generate feature maps or images of the target modality using data from other available modalities when specific modal image data is missing, based on a generative adversarial network architecture.

[0008] Specifically, the multimodal feature fusion and decision-making module includes a feature extraction unit, a graph fusion unit, and an adaptive decision-making unit. The feature extraction unit is used to extract feature vectors or feature maps of different modal data using a deep convolutional neural network. The graph fusion unit is used to construct a heterogeneous graph from feature nodes of different modalities, component entity nodes, and preset defect concept nodes, and to generate a global fusion feature representation through message passing and feature aggregation via a heterogeneous graph neural network. The adaptive decision-making unit is used to model the detection process as a sequential decision-making process, and dynamically select subsequent operations based on the global fusion feature representation and the current detection context. The operations include: performing another modality detection, focusing on a specific area for fine scanning, performing defect classification, or requesting external intervention.

[0009] Specifically, the adaptive decision-making unit is trained using a reinforcement learning algorithm. Its state space includes the global fusion feature representation, the history of executed detection actions, and information on identified suspicious regions. Its reward function is designed to encourage accurate classification and punish unnecessary resource consumption and incorrect judgments.

[0010] Specifically, the multimodal data acquisition module includes a high-resolution optical camera, an oscilloscope or dedicated testing equipment for acquiring electrical signal parameters, and at least two of an X-ray imager or an ultrasonic scanning microscope.

[0011] A method for testing electronic components includes the following steps:

[0012] S1: Acquire at least two different types of data of the electronic component to be inspected through the multimodal data acquisition module. S2: Evaluate the acquired data through the image enhancement and generation module, and enhance or generate cross-modal images with poor quality or missing data. S3: Extract the features of each modality data through the multimodal feature fusion and decision module, perform deep fusion to obtain a global feature representation, and make a dynamic decision on the next detection action based on this. S4: Repeat some or all of steps S1 to S3 until the adaptive decision unit makes a final defect classification decision or triggers manual review. S5: Output a visual report containing defect type, location, confidence level and detection path traceability through the result display and control module, and control the actuator to re-inspect a specific area.

[0013] Specifically, in step S2, the enhancement of image data involves: performing bilateral filtering on the image to smooth non-edge regions, using the Roberts operator for edge extraction and marking, and in non-edge regions, calculating local variance and selecting pixel values ​​from the region with the largest variance in multiple images for fusion to improve detail clarity.

[0014] Specifically, in step S3, the step of making the next detection action based on this dynamic decision is as follows: the current global fusion features, the sequence of actions already executed, and the environmental state are used as state inputs to a pre-trained reinforcement learning policy network, the network outputs a probability distribution of different detection actions, and the action is selected and executed according to the distribution.

[0015] Specifically, in step S3, the deep fusion is performed by: constructing a heterogeneous graph containing modal feature nodes, component entity nodes, and defect concept nodes; using a heterogeneous graph neural network to update node features through a relationship-specific message passing and aggregation mechanism; wherein the final features of the component entity nodes are used as the global fusion feature representation.

[0016] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: The electronic component inspection system and method of the present invention effectively cope with the working conditions of poor data quality or temporary failure of a single sensor through image enhancement and cross-modal generation technology, ensuring the continuity of the inspection process and the integrity of decision information. Furthermore, by using heterogeneous graph neural networks for multimodal feature fusion, it can model and utilize the complex nonlinear relationships and semantic associations between different modal features, thereby improving the accuracy of defect identification. Moreover, the final report not only includes defect classification and location, but also records the complete inspection decision path and key data, which facilitates quality traceability, model optimization and manual review.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0019] Figure 1 This is a schematic diagram of the electronic component testing system and method of the present invention;

[0020] Figure 2 This is a schematic diagram of the electronic component testing system and method of the present invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention. Rather, embodiments of the invention include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0022] The electronic component testing system and method of the present invention will be described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, the electronic component testing system of this invention includes a multimodal data acquisition module, an image enhancement and generation module, a multimodal feature fusion and decision-making module, and a result display and control module.

[0024] The system includes a multimodal data acquisition module, which synchronously or asynchronously acquires at least two different types of data from the electronic components under test. It utilizes a CCD industrial camera, oscilloscope or integrated circuit tester, micro-focus X-ray detector, and ultrasonic scanning microscope to obtain the relevant data. The data types include optical image data, electrical signal waveform data, and internal structure imaging data. An image enhancement and generation module, electrically connected to the multimodal data acquisition module, is used to assess the quality of the acquired image data and enhance or generate images with poor quality or missing data based on other modal data. A multimodal feature fusion and decision module, electrically connected to both the image enhancement and generation module and the multimodal data acquisition module, is used to extract and fuse features from different modal data, perform defect detection and classification based on the fused features, and dynamically adjust the detection strategy. A result display and control module, electrically connected to the multimodal feature fusion and decision module, is used to display the detection results, provide an interactive interface, and control the system to perform refined re-inspection actions.

[0025] In one embodiment of the present invention, the image enhancement and generation module includes a quality assessment unit, a data enhancement unit, and a cross-modal generation unit.

[0026] The system includes a quality assessment unit for evaluating the sharpness, contrast, or integrity of image data; a data augmentation unit for enhancing poor-quality image data based on bilateral filtering, edge enhancement, and detail fusion algorithms; and a cross-modal generation unit for generating feature maps or images of the target modality using data from other available modalities when specific modal image data is missing, based on a generative adversarial network architecture.

[0027] In one embodiment of the present invention, the multimodal feature fusion and decision module includes a feature extraction unit, a graph fusion unit, and an adaptive decision unit.

[0028] The system includes a feature extraction unit, which uses a deep convolutional neural network to extract feature vectors or feature maps from different modalities; a graph fusion unit, which constructs a heterogeneous graph from feature nodes, component entity nodes, and preset defect concept nodes from different modalities, and generates a global fusion feature representation through message passing and feature aggregation via a heterogeneous graph neural network; and an adaptive decision unit, which models the detection process as a sequential decision-making process and dynamically selects subsequent operations based on the global fusion feature representation and the current detection context. These operations include: performing another modality of detection, focusing on a specific area for fine scanning, performing defect classification, or requesting external intervention.

[0029] In one embodiment of the present invention, the adaptive decision unit is trained using a reinforcement learning algorithm. Its state space includes a global fusion feature representation, the history of executed detection actions, and information on identified suspicious regions. Its reward function is designed to encourage accurate classification and punish unnecessary resource consumption and incorrect judgments.

[0030] In one embodiment of the present invention, the multimodal data acquisition module includes a high-resolution optical camera, an oscilloscope or dedicated testing equipment for acquiring electrical signal parameters, and at least two of an X-ray imager or an ultrasonic scanning microscope.

[0031] like Figure 2 As shown, the electronic component testing method of this invention includes the following steps:

[0032] S1: Acquire at least two different types of data from the electronic component to be inspected through the multimodal data acquisition module. S2: Evaluate the acquired data through the image enhancement and generation module, and enhance or generate cross-modal images with poor quality or missing data. S3: Extract features from each modality of data through the multimodal feature fusion and decision module, perform deep fusion to obtain a global feature representation, and make a dynamic decision on the next detection action based on this. S4: Repeat some or all of steps S1 to S3 until the adaptive decision unit makes a final defect classification decision or triggers manual review. S5: Output a visual report containing defect type, location, confidence level, and detection path traceability through the result display and control module, and control the actuator to re-inspect a specific area.

[0033] In one embodiment of the present invention, step S2 involves enhancing the image data, specifically:

[0034] Bilateral filtering is applied to the image to smooth non-edge regions.

[0035] The Roberts operator is used for edge extraction and labeling.

[0036] In non-edge regions, the pixel values ​​of the region with the largest variance in multiple images are selected and fused to improve detail clarity.

[0037] In one embodiment of the present invention, in step S3, the next detection action is determined based on this dynamic decision, specifically: the current global fusion features, the sequence of actions already executed, and the environmental state are used as state inputs to a pre-trained reinforcement learning policy network, the network outputs a probability distribution for executing different detection actions, and the action is selected and executed according to the distribution.

[0038] In one embodiment of the present invention, in step S3, deep fusion is performed, specifically by: constructing a heterogeneous graph containing modal feature nodes, component entity nodes and defect concept nodes, and using a heterogeneous graph neural network to update node features through a relationship-specific message passing and aggregation mechanism, wherein the final features of the component entity nodes are used as global fusion feature representations.

[0039] In summary, the electronic component inspection system and method of this invention effectively address the situation of poor data quality or temporary failure of a single sensor by using image enhancement and cross-modal generation technology, ensuring the continuity of the inspection process and the integrity of decision information. Furthermore, by employing heterogeneous graph neural networks for multimodal feature fusion, it can model and utilize the complex nonlinear relationships and semantic associations between different modal features, thereby improving the accuracy of defect identification. Moreover, the final report not only includes defect classification and location but also records the complete inspection decision path and key data, facilitating quality traceability, model optimization, and manual review.

[0040] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0041] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An electronic component testing system, characterized in that, It includes a multimodal data acquisition module, an image enhancement and generation module, a multimodal feature fusion and decision-making module, and a result display and control module. A multimodal data acquisition module is used to acquire at least two different types of data from the electronic component under test synchronously or asynchronously, wherein the data types include optical image data, electrical signal waveform data, and internal structure imaging data; The image enhancement and generation module is electrically connected to the multimodal data acquisition module and is used to evaluate the quality of the acquired image data and enhance or generate image data with poor quality or missing data based on other modal data. The multimodal feature fusion and decision module is electrically connected to the image enhancement and generation module and the multimodal data acquisition module. It is used to extract and fuse features from different modal data, perform defect detection and classification based on the fused features, and dynamically adjust the detection strategy. The result display and control module is electrically connected to the multimodal feature fusion and decision module, and is used to display the detection results, provide an interactive interface, and control the system to perform refined re-inspection actions.

2. The electronic component testing system according to claim 1, characterized in that, The image enhancement and generation module includes a quality assessment unit, a data enhancement unit, and a cross-modal generation unit, wherein... A quality assessment unit is used to evaluate the sharpness, contrast, or integrity of image data. The data augmentation unit is used to enhance poor-quality image data based on bilateral filtering, edge enhancement, and detail fusion algorithms; Cross-modal generation units are used to generate feature maps or images of a target modality by utilizing data from other available modalities when image data of a specific modality is missing, based on a generative adversarial network architecture.

3. The electronic component testing system according to claim 1, characterized in that, The multimodal feature fusion and decision-making module includes a feature extraction unit, a graph fusion unit, and an adaptive decision-making unit, wherein... The feature extraction unit is used to extract feature vectors or feature maps of different modalities using a deep convolutional neural network. The graph fusion unit is used to construct a heterogeneous graph by combining feature nodes of different modalities, component entity nodes and preset defect concept nodes, and to generate a global fusion feature representation by performing message passing and feature aggregation through a heterogeneous graph neural network. An adaptive decision unit is used to model the detection process as a sequential decision process. Based on the global fusion feature representation and the current detection context, it dynamically selects subsequent operations, including: performing another modality detection, focusing on a specific area for fine scanning, performing defect classification, or requesting external intervention.

4. The electronic component testing system according to claim 1, characterized in that, The adaptive decision-making unit is trained using a reinforcement learning algorithm. Its state space includes the global fusion feature representation, the history of executed detection actions, and information on identified suspicious regions. Its reward function is designed to encourage accurate classification and punish unnecessary resource consumption and incorrect judgments.

5. The electronic component testing system according to claims 1-4, characterized in that, The multimodal data acquisition module includes a high-resolution optical camera, an oscilloscope or dedicated testing equipment for acquiring electrical signal parameters, and at least two of an X-ray imager or an ultrasonic scanning microscope.

6. A method for testing electronic components, applied to the electronic component testing system according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Acquire at least two different types of data from the electronic component to be inspected through the multimodal data acquisition module; S2: Evaluate the data acquired in step S1 through the image enhancement and generation module, and enhance or generate cross-modal data for poor-quality or missing images; S3: Extract features from each modality of data through the multimodal feature fusion and decision module, perform deep fusion to obtain a global feature representation, and make a dynamic decision on the next detection action based on this; S4: Repeat some or all of steps S1 to S3 until the adaptive decision unit makes a final defect classification decision or triggers manual review; S5: Output a visual report containing defect type, location, confidence level, and detection path tracing through the result display and control module, and control the actuator to re-inspect a specific area.

7. The method for testing electronic components according to claim 6, characterized in that, In step S2, the enhancement of the image data specifically includes: Apply bilateral filtering to the image to smooth non-edge regions; The Roberts operator is used for edge extraction and labeling; In non-edge regions, the pixel values ​​of the region with the largest variance in multiple images are selected and fused to improve detail clarity.

8. The method for testing electronic components according to claim 6, characterized in that, In step S3, the step of making the next detection action based on this dynamic decision is specifically as follows: the current global fusion features, the sequence of actions already executed, and the environmental state are used as state inputs to a pre-trained reinforcement learning policy network, the network outputs a probability distribution of different detection actions, and the action is selected and executed according to the distribution.

9. The method for testing electronic components according to claim 6, characterized in that, In step S3, the deep fusion is specifically performed by: constructing a heterogeneous graph containing modal feature nodes, component entity nodes, and defect concept nodes, and using a heterogeneous graph neural network to update node features through a relationship-specific message passing and aggregation mechanism, wherein the final features of the component entity nodes are used as the global fusion feature representation.

Images