Electronic component assembling system based on multi-mode perception and intelligent decision

Abstract

The invention belongs to the field of industrial automation, and provides an electronic component assembling system based on multi-mode perception and intelligent decision making. By integrating a multi-mode sensor and applying technologies such as multi-mode sensor fusion, real-time monitoring and dynamic regulation and control, a knowledge graph, force sense feedback and digital twinning and virtual debugging, high-precision positioning of elements, electrostatic protection, self-adaptive optimization of an assembly process, precise assembly control and simulation and improvement of an assembly process are realized; the problems of positioning, electrostatic protection, process adaptability and the like of an industrial robot in precise electronic component assembling are solved.

Description

Technical Field

[0001] The present invention relates to the field of industrial robot automation, and particularly to an electronic component assembly system based on multi-modal perception and intelligent decision-making. Background Art

[0002] Semiconductor wafer manufacturing is the core link of the semiconductor industry, with complex processes and extremely high requirements for precision and quality. Industrial robots play an important role in various processes of semiconductor wafer manufacturing due to their automation, high precision, and stability. However, with the continuous progress of semiconductor technology and the increasing refinement of chip manufacturing processes, a series of problems have emerged in the existing industrial robot technology during wafer manufacturing, severely restricting the improvement of the efficiency and quality of semiconductor wafer manufacturing.

[0003] Problems of the Existing Technology

[0004] Insufficient positioning accuracy and stability during wafer handling: In semiconductor wafer manufacturing, wafers need to be frequently handled between different processing equipment. Traditional industrial robot handling systems mainly rely on simple visual positioning and preset motion trajectories, and cannot accurately adapt to factors such as the minute dimensional deviations of wafers, the slight changes in placement positions, and vibrations during handling. This results in the wafers being prone to offset and collision during handling, not only affecting the positioning accuracy of the wafers, but also potentially causing physical damage to the wafers, reducing the product qualification rate and production efficiency.

[0005] Low efficiency and poor accuracy in detecting wafer surface defects: Defects on the wafer surface will directly affect the performance and reliability of the chips, so the detection of wafer surface defects is crucial. Existing industrial robot detection systems mainly rely on single visual detection technology and simple image processing algorithms, and cannot accurately identify various types and sizes of surface defects, such as minute scratches, particle contamination, crystal defects, etc. Moreover, due to the slow detection speed, it is difficult to meet the detection requirements of large-scale wafer production, and situations of missed detection and false detection are likely to occur, affecting the control of product quality.

[0006] Low precision in controlling the thin film deposition process: Thin film deposition is one of the key processes in semiconductor wafer manufacturing, and its purpose is to deposit a uniform and high-quality thin film on the wafer surface. Existing thin film deposition systems controlled by industrial robots mainly adopt fixed process parameters and preset deposition paths, and cannot be adjusted in real time according to the surface characteristics of the wafers, temperature changes, and real-time situations during deposition. This results in uneven thickness and inconsistent composition of the thin film deposition, affecting the performance and reliability of the chips and increasing production costs. Summary of the Invention

[0007] The present invention provides an electronic component assembly system based on multi-modal perception and intelligent decision-making, including:

[0008] The high-precision component positioning algorithm module based on multi-modal sensor fusion integrates various types of sensors such as high-resolution vision sensors, laser range sensors, and capacitive sensors on industrial robots to obtain multi-modal information such as images, distances, and capacitances of components in real time. Using multi-modal sensor fusion technologies such as fusion networks based on attention mechanisms and combining deep learning algorithms such as convolutional neural networks (CNNs), it realizes high-precision positioning of precision electronic components.

[0009] The electrostatic protection system module based on real-time monitoring and dynamic regulation deploys electrostatic monitoring sensors at key parts of industrial robots (such as end effectors, grippers, etc.) and in the working environment to monitor parameters such as electrostatic field intensity and electrostatic charge amount during the assembly process in real time. Using big data analysis and machine learning algorithms such as prediction models based on neural networks, it establishes an electrostatic prediction model to realize real-time monitoring and dynamic regulation of static electricity.

[0010] The assembly process adaptive learning and optimization system module based on knowledge graphs collects and organizes a large amount of assembly process knowledge and experience of different types of precision electronic components, including information such as component types, assembly processes, process parameters, and quality standards. It uses ontology modeling technology to construct an assembly process knowledge graph and uses knowledge graph technology to automatically generate suitable assembly process plans and optimize them according to actual assembly feedback.

[0011] The precise assembly control algorithm module based on force feedback installs high-precision force sensors on the end effector of the industrial robot to monitor the acting force and reaction force during the assembly process in real time. Using force feedback information and combining with the robot's motion control algorithm, it realizes precise control of the assembly process.

[0012] The digital twin and virtual commissioning platform module for the assembly process uses 3D modeling technology and simulation software to establish a digital twin model of the physical assembly production line, simulate and simulate the assembly process in a virtual environment, and synchronously update the virtual model in real time according to actual assembly data to realize continuous improvement and quality improvement of the assembly process.

[0013] Furthermore, in the high-precision component positioning algorithm module based on multi-modal sensor fusion, the data preprocessing after multi-modal data fusion adopts normalization or standardization methods.

[0014] Furthermore, in the electrostatic protection system module based on real-time monitoring and dynamic regulation, the electrostatic protection measures also include setting up an anti-static shielding cover in the working area.

[0015] Furthermore, in the knowledge graph-based assembly process adaptive learning and optimization system module, when the knowledge graph is updated, an incremental learning algorithm is used to process and integrate the newly added assembly process knowledge.

[0016] Furthermore, in the force-feedback-based precision assembly control algorithm module, when adjusting the motion trajectory according to the force-feedback information, a path planning algorithm such as the A* algorithm is used for trajectory optimization.

[0017] Furthermore, in the digital twin and virtual commissioning platform module for the assembly process, in virtual commissioning, the Monte Carlo simulation method is used to evaluate the reliability of different assembly schemes.

[0018] Furthermore, in the high-precision component positioning algorithm module based on multi-modal sensor fusion, when deploying sensors, the component size, shape, and ambient light conditions of the working environment are considered to ensure comprehensive and accurate component information is obtained.

[0019] Furthermore, in the electrostatic protection system module based on real-time monitoring and dynamic regulation, the sampling frequency of the electrostatic monitoring sensor is dynamically adjusted according to the electrostatic generation frequency of the assembly process.

[0020] Furthermore, in the knowledge graph-based assembly process adaptive learning and optimization system module, when generating an assembly process plan, multi-objective optimization factors such as component cost, production efficiency, and quality requirements are considered.

[0021] Furthermore, in the force-feedback-based precision assembly control algorithm module, the accuracy of the force sensor is ±0.01 N, which can meet the high-precision requirements for force control in the assembly of precision electronic components.

[0022] Beneficial effects:

[0023] The key technologies of this system significantly improve the level of precision electronic component assembly. The high-precision component positioning algorithm based on multi-modal sensor fusion breaks through the limitations of traditional single-vision positioning, integrates multi-source information, accurately identifies the position and posture of components, improves the positioning accuracy from ±0.2 mm to below ±0.05 mm, and reduces the component assembly offset rate from 12% to below 3%, greatly improving the assembly quality, reducing rework, and enhancing production efficiency.

[0024] The electrostatic protection system based on real-time monitoring and dynamic regulation comprehensively monitors electrostatic parameters, predicts risks in advance, and automatically adjusts protection measures, reducing the component electrostatic damage rate from 7% to below 2% and increasing the product qualification rate from 90% to 96%, effectively guaranteeing production benefits.

[0025] The assembly process adaptive learning and optimization system based on knowledge graph breaks through the limitations of traditional design for specific processes, quickly generates and optimizes assembly process plans, changes from manual adjustment that takes 2 - 3 days to automatic completion within a few hours, increases production efficiency by about 60%, and adapts to complex and changeable assembly requirements.

[0026] The precise assembly control algorithm based on force feedback perceives the assembly force in real time, accurately adjusts the motion trajectory and force, reduces the component damage rate from 5% to below 1%, and reduces the insecure assembly rate from 8% to below 3%, greatly enhancing the assembly stability and product reliability.

[0027] The digital twin and virtual commissioning platform for the assembly process bids farewell to the traditional way that relies on experience and actual testing, discovers more than 90% of potential problems in advance, shortens the new product R & D cycle from 3 - 4 months to within 2 months, increases production efficiency by about 45%, effectively ensures the stability of product quality, reduces R & D costs and risks, and comprehensively promotes the precision electronic component assembly industry towards a new stage of high efficiency, precision, and intelligence. Description of the Drawings

[0028] Figure 1 Overall process schematic diagram. Detailed Implementation Modes

[0029] Example 1

[0030] Example of a high-precision component positioning algorithm based on multi-modal sensor fusion

[0031] Sensor deployment and data acquisition: On the end effector of an industrial robot, a high-resolution vision sensor with a resolution of 2500×2000 is installed horizontally to ensure obtaining high-definition images of components and accurately identifying their positions and postures; a capacitive sensor is installed near the grasping part to detect whether the component is approaching and preliminarily judge the position, with an accuracy of up to ±0.03mm; a laser ranging sensor is installed in the key area covered by the robotic arm, with a measurement range of 0 - 100mm and an accuracy of ±0.02mm, to monitor the distance between the end effector and the component in real time. Each sensor collects data at a frequency of 120Hz and transmits it to the data processing unit through a high-speed Ethernet to ensure the stability and efficiency of data transmission.

[0032] Multi-modal data fusion and state model construction: The data processing unit uses a fusion network based on the attention mechanism. First, preprocess the visual image by edge detection, feature point extraction, etc., then perform convolution operations to extract key features such as the contour and shape of the component; filter and denoise the data of the capacitive sensor to remove environmental interference; normalize the laser ranging data to unify the dimension. The fusion network constructs a real-time state model of the component position and posture by learning the weights of different modal data, highlighting key information and improving the positioning accuracy.

[0033] Component positioning and identification: Using a deep learning algorithm based on convolutional neural network (CNN) to analyze the fused data. CNN contains multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layers extract local features of components through convolutional kernels of different sizes. The pooling layers perform downsampling to reduce the amount of data and retain key features. The fully connected layers map the extracted features to specific position and pose information to achieve high-precision positioning of precision electronic components, with a positioning accuracy of up to

[0034] ±0.05 mm.

[0035] Modeling and solving process: In the CNN training stage, the cross-entropy loss function is used to measure the difference between the predicted position and the true position. The Adam optimizer is used, with a learning rate set to 0.0008, and iterative training is performed 1200 times. Through the backpropagation algorithm, the parameters such as the weights and biases of the convolutional kernels in the network are continuously adjusted to continuously reduce the prediction error of the model for the component position and pose until convergence.

[0036] Efficiency improvement: In the traditional single vision positioning method, the positioning accuracy error is about ±0.2 mm. In the case of complex lighting and component reflection, the positioning error is even larger, resulting in a component assembly deviation rate of 12%. The positioning accuracy error of this algorithm is reduced to

[0037] below ±0.05 mm, the component assembly deviation rate is reduced to below 3%, the assembly quality is significantly improved, the rework rate is greatly reduced, and the production efficiency is increased by about 40%. In actual production, the number of components assembled per hour is increased from 200 to 280, and due to the improvement of assembly accuracy, the product qualification rate is increased from 85% to 95%.

[0038] Embodiment of an electrostatic protection system based on real-time monitoring and dynamic regulation

[0039] Deployment of electrostatic monitoring sensors and data collection: Electrostatic field intensity sensors and electrostatic charge sensors are evenly deployed at key parts such as the end effector of the industrial robot, the component transfer track, and the workbench, as well as in the working environment. The accuracy of the electrostatic field intensity sensor is ±5 V / m, which can monitor the electrostatic field intensity in space in real time; the accuracy of the electrostatic charge sensor is

[0040] ±0.01 pC, which can accurately detect the electrostatic charge on the surface of components and equipment. The sensors collect data in real time at a frequency of 50 Hz and transmit the data to the electrostatic data analysis module through a wireless transmission module to ensure the real-time nature of the data.

[0041] Establishment of Electrostatic Data Analysis and Prediction Model: Use a prediction model based on neural network to analyze the collected electrostatic data. First, normalize the data to eliminate the influence of dimension. The neural network adopts a multi-layer perceptron structure, including an input layer, multiple hidden layers, and an output layer. The input layer receives the normalized electrostatic data, the hidden layers perform feature extraction and transformation through non-linear activation functions, and the output layer predicts the electrostatic risk level in the future for a period of time, such as low, medium, and high. Through training with a large amount of historical data, continuously adjust the weights and biases of the neural network to make the prediction accuracy of the model reach more than 90%.

[0042] Dynamic Adjustment of Electrostatic Protection Measures: When it is predicted that the electrostatic risk is at the intermediate level, the system automatically activates the ion air gun and adjusts the air volume and wind speed to reduce the electrostatic field strength in the working area; when the risk reaches the high level, in addition to activating the ion air gun, the grounding resistance is also dynamically adjusted to enhance the electrostatic discharge ability and ensure that electronic components are not damaged by static electricity during the assembly process.

[0043] Modeling and Solving Process: In the neural network training, use the mean square error loss function, adopt the stochastic gradient descent (SGD) algorithm, set the learning rate to 0.01, the momentum to 0.9, and iterate and train 800 times. By continuously adjusting the weights and biases, minimize the error between the predicted value of the model and the actual electrostatic risk value.

[0044] Efficiency Enhancement: In the traditional simple grounding and anti-static workbench methods, the component damage rate caused by static electricity is about 7%. This system reduces the component damage rate to less than 2%, and the product qualification rate is increased from 90% to 96%, effectively reducing the economic losses caused by static electricity problems and improving production efficiency.

[0045] Embodiment of an Adaptive Learning and Optimization System for Assembly Process Based on Knowledge Graph

[0046] Collection and Arrangement of Assembly Process Knowledge: Collect the assembly process knowledge of different types of precision electronic components, such as resistors, capacitors, chips, etc., including process parameters such as welding temperature, welding time, adhesive selection, curing time, etc., as well as the assembly quality standards and common problem-solving methods of different components in different environments. Classify, label, and structure the collected knowledge to prepare for constructing the knowledge graph.

[0047] Construction and Update of Knowledge Graph: Adopt ontology modeling technology to construct the sorted assembly process knowledge into a knowledge graph. In the knowledge graph, component types, assembly processes, process parameters, etc. are used as nodes, and the association relationships between them are used as edges. For example, the "chip" node is connected to process parameter nodes such as "welding temperature" and "welding time" through the "welding process" edge. During the actual assembly process, if new assembly processes or problem-solving solutions appear, update the knowledge graph in a timely manner to ensure the timeliness and integrity of the knowledge.

[0048] Assembly process plan generation and optimization: When encountering a new assembly task, the system uses knowledge graph technology to search for and match relevant assembly process knowledge in the knowledge graph based on information such as the type, quantity, and quality requirements of the input components, and automatically generates a preliminary assembly process plan. Then, according to the feedback data during the actual assembly process, such as the results of welding quality inspection and the firmness of component bonding, optimization algorithms are used to optimize and adjust the process plan to improve assembly efficiency and quality.

[0049] Modeling and solving process: In the optimization algorithm, a genetic algorithm is used to optimize the process parameters. The process parameters are encoded as chromosomes, and through genetic operations such as selection, crossover, and mutation, continuous iteration and optimization are carried out to find the combination of process parameters that maximizes assembly efficiency and quality. For example, in the optimization of welding process parameters, the welding temperature, time, and other parameters are adjusted through the genetic algorithm to reduce the welding defect rate.

[0050] Efficiency improvement: Traditional assembly systems designed for specific processes require manual adjustment and reprogramming when changing assembly products or adopting new assembly processes, which takes about 2 - 3 days. This system can generate and optimize the assembly process plan based on the knowledge graph within a few hours, increasing the production efficiency by about 60%, and can adapt to a variety of complex assembly processes, reducing labor costs and production cycles.

[0051] Embodiment of a precise assembly control algorithm based on force feedback

[0052] Force sensor installation and data acquisition: A high-precision six-axis force sensor is installed on the end effector of the industrial robot, which can simultaneously measure the forces in the X, Y, and Z directions and the torques around these three axes, with a measurement accuracy of ±0.01N and

[0053] ±0.001N·m. The force sensor collects the acting and reacting force data during the assembly process in real time at a frequency of 80Hz and transmits it to the force feedback analysis module through a dedicated data transmission line to ensure the accuracy and real-time nature of the data.

[0054] Force feedback analysis and control strategy generation: The force feedback analysis module performs preprocessing such as filtering and noise reduction on the data collected by the force sensor, and extracts force feedback features, such as the change trend of force magnitude and the fluctuation of torque. The precise assembly control module generates a control strategy based on the force feedback features in combination with the robot's motion control algorithm. For example, during the component insertion process, when the force sensor detects that the insertion force exceeds the set threshold, the control strategy is to reduce the insertion speed or adjust the insertion angle to avoid component damage or loose assembly.

[0055] Precision assembly control execution: The industrial robot adjusts the motor speed, torque and other parameters according to the control strategy generated by the precision assembly control module, and adjusts the motion trajectory and force in real time. In the welding process, the pressure and moving speed of the welding head are adjusted in real time according to the force feedback to ensure stable and reliable welding quality.

[0056] Modeling and solving process: When generating the control strategy, the model predictive control (MPC) algorithm is adopted. According to the force feedback data and the robot motion model, the force and motion states in the future period are predicted. By optimizing the objective function, such as minimizing the force deviation and making the motion smooth, the optimal control input, that is, the control parameters of the motor, is calculated.

[0057] Efficiency improvement: In the traditional assembly method without force feedback control, the component damage rate reaches 5% and the insecure assembly rate is 8% due to improper force application. This algorithm reduces the component damage rate to less than 1% and the insecure assembly rate to less than 3%. The assembly quality and stability are greatly improved, the product reliability is enhanced, and the market competitiveness is improved.

[0058] Embodiment of digital twin and virtual commissioning platform for the assembly process

[0059] Digital twin model construction: Using professional 3D modeling software, combined with the data of physical entities such as industrial robots, assembly equipment, and electronic components collected by sensors in real time, a digital twin model of the precision electronic component assembly process is established. The model includes the kinematic model and dynamic model of the robot, the working principle model of the assembly equipment, and the physical property model of the electronic component, etc., to realize the real-time interaction and synchronization between the physical entity and the virtual model, and ensure that the virtual model can accurately reflect the actual assembly process.

[0060] Virtual commissioning and optimization: In the virtual environment, simulate and verify different assembly schemes and process parameters. Set different parameter combinations such as robot motion speed, trajectory, welding temperature, time, and adhesive application amount, and analyze the impact of different schemes on assembly efficiency and quality. Use optimization algorithms, such as particle swarm optimization algorithm, to find the optimal assembly scheme and process parameter combination. For example, through simulation, it is found that the welding defect rate is the lowest under a certain combination of welding temperature and time, and this parameter is applied to actual production.

[0061] Real-time synchronization and continuous improvement: In the actual assembly process, the digital twin model and the physical production line are synchronized through real-time data transmission. According to the actual assembly data, such as component positioning deviation, force feedback data, welding quality inspection data, etc., the virtual model is updated and optimized. By continuously comparing and analyzing the data of the virtual model and the physical production line, potential problems are found and improved in time to achieve continuous optimization and quality improvement of the assembly process.

[0062] Modeling and solution process: In the particle swarm optimization algorithm, the assembly process parameters are regarded as particles, and each particle represents an assembly scheme. The particles search for the optimal solution in the solution space. By continuously updating the velocity and position of the particles, the particles approach the optimal solution, and finally find the process parameter combination that optimizes the assembly efficiency and quality.

[0063] Efficiency improvement: In the traditional way of optimizing the assembly process by experience and actual testing, the R & D cycle of new products is long, about 3 - 4 months, and it is difficult to discover potential problems. This platform can discover more than 90% of potential problems in advance through virtual commissioning, shorten the R & D cycle of new products to within 2 months, increase the production efficiency by about 45%, significantly improve the product quality stability, and reduce the R & D cost and production risk.

Claims

1. An electronic component assembly system based on multimodal perception and intelligent decision-making, characterized in that: include: The high-precision component positioning algorithm module based on multi-modal sensor fusion integrates high-resolution visual sensors, laser ranging sensors, capacitive sensors and other types of sensors on industrial robots to obtain the component's image, distance, and capacitance multi-modal information in real time. It uses multi-modal sensor fusion technology combined with deep learning algorithms to achieve high-precision positioning of precision electronic components. Based on the electrostatic protection system module of real-time monitoring and dynamic control, electrostatic monitoring sensors are deployed in the key parts and working environment of industrial robots to monitor the electrostatic field strength and electrostatic charge parameters in real time during the assembly process. By using big data analysis and machine learning algorithms, an electrostatic prediction model is established to achieve real-time monitoring and dynamic control of static electricity. The assembly process adaptive learning and optimization system module based on knowledge graph collects and organizes a large amount of different types of precision electronic component assembly process knowledge and experience, including component type, assembly process, process parameters, quality standards and other information, and uses ontology modeling technology to build an assembly process knowledge graph. Using knowledge graph technology, it automatically generates a suitable assembly process plan and optimizes it according to actual assembly feedback; The precision assembly control algorithm module based on force feedback installs a high-precision force sensor on the end effector of the industrial robot to monitor the action and reaction forces in the assembly process in real time, and uses the force feedback information in combination with the robot's motion control algorithm to achieve precise control of the assembly process. The digital twin and virtual debugging platform module of the assembly process uses 3D modeling technology and simulation software to establish a digital twin model of the physical assembly production line, simulate and emulate the assembly process in a virtual environment, and synchronously update the virtual model in real time according to the actual assembly data, so as to achieve continuous improvement and quality improvement of the assembly process.

2. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: In the high-precision component positioning algorithm module based on multimodal sensor fusion, the data preprocessing after multimodal data fusion adopts a normalization or standardization method.

3. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: In the electrostatic protection system module based on real-time monitoring and dynamic regulation, the electrostatic protection measures also include setting up an anti-static shielding cover in the working area.

4. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: The knowledge graph-based assembly process adaptive learning and optimization system module uses an incremental learning algorithm to process and integrate newly added assembly process knowledge when the knowledge graph is updated.

5. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: In the precision assembly control algorithm module based on force feedback, when adjusting the motion trajectory according to the force feedback information, a path planning algorithm such as the A* algorithm is used to optimize the trajectory.

6. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: The digital twin and virtual debugging platform module of the assembly process uses a Monte Carlo simulation method to evaluate the reliability of different assembly schemes during virtual debugging.

7. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: The high-precision component positioning algorithm module based on multi-modal sensor fusion takes into account the component size, shape and working environment lighting conditions when deploying sensors to ensure that comprehensive and accurate component information is obtained.

8. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: In the electrostatic protection system module based on real-time monitoring and dynamic regulation, the sampling frequency of the electrostatic monitoring sensor is dynamically adjusted according to the frequency of static electricity generation in the assembly process.

9. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1 is characterized in that: The knowledge graph-based assembly process adaptive learning and optimization system module considers multi-objective optimization factors such as component cost, production efficiency, and quality requirements when generating an assembly process plan.

10. The industrial robot precision electronic component assembly system based on multimodal perception and intelligent decision-making according to claim 1, characterized in that: The precision assembly control algorithm module based on force feedback has a force sensor with an accuracy of ±0.01N, which can meet the high-precision requirements of force control for precision electronic component assembly.

Images