Detection method and device for steering wheel frame detection assembly line and readable storage medium thereof

Abstract

The invention provides a detection method and device for a steering wheel frame detection assembly line and a readable storage medium, and the method comprises the steps: obtaining multi-modal data of a steering wheel frame at a welding station, including a multi-angle welding part image, distance data between parts, and force change data; after multi-modal data is preprocessed, modal features are extracted through a multi-branch deep learning model, assembly quality classification or regression prediction is carried out through a full connection layer after fusion, and a detection model is trained; a reinforcement learning agent is constructed, assembly line control parameters serve as action space, assembly quality evaluation indexes serve as reward signals, and an optimal control parameter combination is selected according to the multi-modal data and the assembly line state represented by the detection result; a detection result is fed back, related data are stored, and the assembly line control system performs self-adaptive adjustment according to the data. The detection accuracy can be effectively improved, assembly line control is optimized, the assembly quality and production efficiency of the automobile steering wheel frame are improved, and intelligent manufacturing development is promoted.

Description

Technical Field

[0001] The present invention relates to the field of mechanical automation technology, and in particular to a detection method and device for a steering wheel frame detection pipeline and a readable storage medium thereof. Background Art

[0002] In the field of automobile manufacturing, the steering wheel is a component that the driver directly controls, and the assembly quality of its frame plays a decisive role in the safety, comfort and handling performance of the car. With the rapid development of the automobile industry, consumers have increasingly stringent requirements for automobile quality, which has prompted automobile manufacturers to continuously seek more efficient and accurate assembly inspection technologies to ensure the consistency and stability of product quality.

[0003] Traditional automobile steering wheel frame assembly inspection mainly relies on manual inspection methods. Inspectors use visual observation and simple measuring tools to check the appearance of welding parts, the assembly integrity of parts, etc. This method has many defects that are difficult to overcome. First, manual inspection is inefficient and cannot meet the needs of large-scale and efficient production of modern automobiles. Inspectors are prone to fatigue after long hours of work, which slows down the inspection speed and makes it difficult to ensure that every product can be carefully inspected, which may miss some potential quality problems. Second, the accuracy of inspection is greatly affected by the subjective factors of the inspectors. Different inspectors have different experiences, skill levels and concentration levels, and the judgment criteria for the same defect may be different, which can easily lead to misjudgment and omission, seriously affecting the stability of product quality. Third, manual inspection is difficult to accurately identify complex assembly features and minor defects. The structure of the automobile steering wheel frame is becoming more and more complex. Some minor welding defects and minor assembly deviations between parts are difficult to detect by manual visual inspection alone, so these problems cannot be dealt with in time, increasing the risk of unqualified products entering the market.

[0004] In recent years, with the continuous advancement of automation technology, some automated testing equipment has begun to be used in the field of automotive steering wheel frame assembly testing. These devices have improved the detection efficiency to a certain extent, but there are still limitations. For example, detection equipment based on a single sensor can only obtain information on a certain aspect of the product and cannot comprehensively and accurately evaluate the assembly quality. For some complex defects that require comprehensive consideration of multiple factors, single sensor detection is often powerless. Moreover, most of the existing automated testing equipment lacks adaptive adjustment capabilities and cannot optimize the process parameters in the production process in real time based on the test results, resulting in the inability to make timely adjustments once quality problems occur in the production process, thus affecting the production efficiency and product quality of the entire production line.

[0005] Deep learning technology has achieved great success in computer vision, pattern recognition and other fields, bringing new ideas and methods to industrial inspection. Deep learning models can automatically learn and extract complex features in images and data, and have strong classification and prediction capabilities. However, in the assembly inspection of automobile steering wheel frames, single-modal deep learning models (such as inspection models based only on visual images) are difficult to fully utilize multi-source information in the assembly process. The assembly quality of automobile steering wheel frames involves many aspects, including welding quality, component assembly accuracy, mechanical properties, etc. Single-modal data cannot fully reflect these complex assembly states, thus limiting the detection accuracy and reliability of deep learning models in this field.

[0006] At the same time, reinforcement learning, as a technology that can achieve intelligent decision-making and optimized control, has gradually attracted attention in industrial production. However, in the assembly and inspection line of automobile steering wheel frames, the combination of reinforcement learning technology and deep learning technology is still relatively rare. The existing production line lacks an intelligent system that can automatically adjust the control parameters according to the real-time detection results to achieve dynamic optimization of the production process. This leads to the inability of the assembly line to make adaptive adjustments in time when facing various changes and uncertainties in the production process, making it difficult to maximize production efficiency and product quality. For example, our company has applied for patent CN118237903A, a kind of automobile steering wheel frame assembly and inspection line, which only uses CCD detection module and uses visual detection to determine whether the welding parts of the automobile steering wheel frame after welding are qualified. Although it is also mentioned that the welding temperature, pressure and other parameters can be automatically adjusted according to the actual production situation to meet the welding requirements of automobile steering wheel frames of different materials or specifications, there is no specific public means, which is only in the initial conception stage and has not yet been implemented.

[0007] In summary, the field of automobile steering wheel frame assembly and inspection urgently needs an innovative technical method that can integrate multimodal information, give full play to the advantages of deep learning in feature extraction and quality inspection, and combine reinforcement learning to achieve intelligent optimization control of the assembly line, so as to overcome the various shortcomings of the existing technology, improve the assembly quality and production efficiency of automobile steering wheel frames, and meet the growing high-quality and high-efficiency production needs of the automobile manufacturing industry. Summary of the invention

[0008] The embodiments of the present invention provide a detection method, device and readable storage medium for a steering wheel frame detection pipeline, which aims to address the problems of current technologies such as low detection efficiency, poor accuracy, difficulty in comprehensively evaluating assembly quality, lack of adaptive adjustment capabilities, insufficient use of single-modal information of deep learning models, and failure to effectively combine reinforcement learning to achieve intelligent optimization control, and are unable to meet the high-quality and high-efficiency production needs of the automotive manufacturing industry.

[0009] The core technology of this invention is to use a multimodal deep learning model to integrate multi-source data such as vision, ranging, force feedback, etc. to perform precise assembly quality inspection, and use reinforcement learning agents to dynamically optimize assembly line control parameters according to the inspection results. At the same time, it realizes system integration and human-machine collaboration to improve the intelligence level and overall performance of the automobile steering wheel frame assembly inspection line.

[0010] In a first aspect, the present invention provides a detection method for a steering wheel frame detection pipeline, the method comprising the following steps: S00, acquiring multi-angle welding position images of the steering wheel frame of the welding station, distance data between welding parts and force change data during welding operation, and comprehensively obtaining multimodal data; S10, preprocessing the multimodal data; S20, extracting features from the preprocessed multimodal data based on a multi-branch deep learning model, including a visual image branch, a laser ranging data branch, and a force sensor data branch; Extract texture, shape and structural features from images of multimodal data through the visual image branch; Extract the temporal variation characteristics of the distance data of the multi-modal data through the laser ranging data branch; Extract the size and change trend characteristics of the force change data of the multi-modal data through the force sensor data branch; S30, in the fusion layer of the multi-branch deep learning model, the features extracted from each branch are fused, and then a series of fully connected layers are used to perform the final assembly quality classification or regression prediction; The multi-branch deep learning model is deeply trained using sample data covering various normal and defective assembly situations to obtain a detection model that is capable of identifying various types of assembly defects. S40, constructing a reinforcement learning agent, setting the key control parameters of the assembly line as the action space of the agent, using the assembly quality evaluation index as the basis of the reward signal, and the key control parameters are the control parameters of each device on the assembly line; The agent selects the optimal control parameter combination through the reinforcement learning algorithm based on the comprehensive state of the current pipeline to maximize the long-term reward; the comprehensive state of the current pipeline is jointly represented by multimodal data and detection results; S50, immediately feeding back the evaluation results of the assembly quality to the assembly line control system, and storing the multimodal data, the test results and the relevant production information in the database; S60, the assembly line control system adaptively adjusts the production process based on the detection results and the decision instructions of the reinforcement learning agent.

[0011] Furthermore, in step S10, the preprocessing includes: Filtering algorithms are used to remove noise interference, statistical methods are used to identify and process outliers, and timestamps are used to accurately mark different modal data to ensure perfect synchronization of multimodal data in time and space.

[0012] Furthermore, in step S20, the visual image branch adopts a multi-layer convolutional neural network structure; The laser ranging data branch adopts a recurrent neural network structure; The force sensor data branch adopts a multi-layer fully connected neural network.

[0013] Furthermore, in step S30, feature vectors from different branches are integrated by weighted summation or concatenation and fusion.

[0014] Furthermore, in step S30, the back propagation algorithm is used to fine-tune the model parameters, and the training samples are expanded in combination with data enhancement technology to improve the generalization ability of the model.

[0015] Furthermore, in step S40, in constructing the reinforcement learning agent, before setting the action space of the agent, the state space of the agent is first defined, and the state space is based on the feature vector after the current multimodal data is fused and the detailed statistical information of the assembly quality of the previous several detection cycles.

[0016] Furthermore, the method also includes step S70, regularly collecting new sample data from the production line, including various normal and newly appeared defect samples, and performing incremental training on the detection model.

[0017] In a second aspect, the present invention provides a detection device for a steering wheel frame detection line, comprising: The acquisition module is used to obtain multi-angle welding part images of the steering wheel frame of the welding station, the distance data between the welding parts, and the force change data during the welding operation, and comprehensively obtain multi-modal data; A preprocessing module, used for preprocessing multimodal data; A feature extraction module is used to extract features from the preprocessed multimodal data based on a multi-branch deep learning model, including a visual image branch, a laser ranging data branch, and a force sensor data branch; the texture, shape, and structural features in the image of the multimodal data are extracted through the visual image branch; the time series change features of the distance data of the multimodal data are extracted through the laser ranging data branch; and the size and change trend features of the force change data of the multimodal data are extracted through the force sensor data branch; The feature fusion module is used to fuse the features extracted from each branch in the fusion layer of the multi-branch deep learning model, and then perform the final assembly quality classification or regression prediction through a series of fully connected layers; the multi-branch deep learning model is deeply trained using sample data covering various normal and defective assembly situations to obtain a detection model that can be prepared to identify various types of assembly defects; The agent module is used to build a reinforcement learning agent. The key control parameters of the assembly line are set as the action space of the agent. The assembly quality evaluation index is used as the basis for the reward signal. The key control parameters are the control parameters of each device on the assembly line. The agent selects the optimal control parameter combination through reinforcement learning algorithm based on the comprehensive state of the current assembly line to maximize long-term rewards; the comprehensive state of the current assembly line is jointly characterized by multimodal data and detection results; the evaluation results of assembly quality are immediately fed back to the assembly line control system; The assembly line control system is used to store multimodal data, test results and related production information in the database; based on the test results and the decision instructions of the reinforcement learning agent, the production process is adaptively adjusted.

[0018] In a third aspect, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned detection method for the steering wheel frame detection pipeline.

[0019] In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program includes program codes for controlling a process and executing a process, wherein the process includes the detection method for a steering wheel frame detection pipeline according to the above-mentioned method.

[0020] The main contributions and innovations of the present invention are as follows: 1. Improved detection accuracy Multimodal data comprehensively reflects the assembly status, different modal data complement each other, and deep learning models accurately extract features. After fusion and training, they can accurately identify defects and reduce the risk of misjudgment and missed judgment. Compared with traditional and single modal methods, the detection accuracy is higher.

[0021] 2. Improved production efficiency Reinforcement learning agents optimize control parameters in real time to reduce downtime adjustments due to quality issues. The detection and feedback mechanism accelerates product flow, improves the overall production efficiency of the assembly line, and adapts to the needs of large-scale automobile production.

[0022] 3. Enhanced system adaptability and stability The data enhancement of the deep learning model enables it to adapt to various situations, and the reinforcement learning agent copes with uncertainty. The combination of the two ensures that the system operates stably in complex environments, has strong robustness, and reduces maintenance costs. In the present invention, the agent is a key component of the reinforcement learning system. It plays the role of a decision maker and continuously learns and optimizes its decision-making strategy by interacting with the environment (i.e., the assembly and inspection pipeline of the automobile steering wheel frame) to achieve specific goals.

[0023] 4. Promote intelligent manufacturing and human-machine collaboration Realize intelligent detection and optimization, and promote the development of automobile manufacturing towards intelligent manufacturing. The human-machine collaborative interface facilitates personnel monitoring and intervention, and the human experience and intelligent system complement each other, improving the production management level and product quality.

[0024] The details of one or more embodiments of the invention are set forth in the following drawings and description so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of a detection method for a steering wheel frame detection pipeline according to an embodiment of the present invention; Figure 2 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0028] The present invention is based on an improvement of an automobile steering wheel frame assembly and detection production line in CN118237903A. Without modifying the overall frame, the present invention solves the defects of the prior art by adding a high-definition visual camera, a high-precision laser rangefinder, a high-sensitivity force sensor (the specific installation position is not limited, as long as accurate data can be collected, so the hardware structure of the original production line is not repeated), and upgrading the control system so that it is equipped with the detection method of the present invention.

[0029] The core technology of the present invention is to deploy multi-type sensors at key nodes in the production line of automobile steering wheel frames to collect multi-modal data, use a multi-branch deep learning model to fuse features to detect assembly quality, use reinforcement learning agents to optimize control parameters based on detection results, and realize system integration and human-machine collaboration, thereby improving the intelligence level and overall performance of assembly detection.

[0030] Embodiment 1 The present invention aims to propose a detection method for a steering wheel frame detection pipeline, specifically, referring to Figure 1 , the method comprises the following steps: S00, Multimodal data acquisition system construction: At the welding station, high-resolution, high-speed frame rate high-definition visual cameras are installed to capture the welding part images from multiple angles to ensure clear capture of weld details, component surface conditions and other information; at the same time, high-precision laser rangefinders are installed to accurately measure the tiny gaps and relative positional relationships between welded components, providing key data support for welding quality assessment. High-sensitivity force sensors are reasonably equipped throughout the entire assembly line process and installed on related equipment or components to accurately detect force changes during operation and monitor the accuracy and stability of production operations in real time.

[0031] In this embodiment, an industrial-grade high-definition visual camera with high resolution (e.g., 5 million pixels) and high frame rate (e.g., 60 frames per second) is selected and installed above and on the side of the welding station. The angle can be adjusted to ensure full coverage of the welding area. The camera is equipped with professional lighting equipment to ensure clear and shadow-free images.

[0032] The laser distance meter uses a high-precision (measurement accuracy of ±0.05mm) and long-range (0.5-5 meters) model. It is installed on a fixed bracket near the welding station. Through precise calibration, its measuring direction is aligned with the key parts of the welding assembly to ensure that the gap and relative position between the welding parts can be accurately measured.

[0033] The force sensor adopts a high-sensitivity (resolution 0.01N) and fast dynamic response model, which is installed in the welding gun fixture of the welding equipment and the key transmission parts of the conveying equipment. The force sensor is connected to the data acquisition card to ensure stable and accurate collection of force data.

[0034] The computing equipment is a server equipped with multiple high-performance GPUs (such as NVIDIA Tesla V100), large-capacity memory (such as 128GB) and high-speed storage (such as 2TBSSD) to run deep learning models and reinforcement learning agent algorithms. The server is connected to the data acquisition unit and the pipeline control system via Gigabit Ethernet to ensure high-speed and stable data transmission. Compared with the cost of several million for the entire pipeline, the budget for these devices is not high, and long-term use can significantly improve detection efficiency and accuracy.

[0035] Preferably, the software environment is: Deep learning model development is based on the TensorFlow framework, which uses its rich function libraries and tools for model building, training, and optimization.

[0036] The OpenAI Gym environment and Ray RLlib library are used for reinforcement learning agent development to achieve agent training, evaluation, and policy optimization.

[0037] The data acquisition software is written in C++ and realizes the synchronous acquisition, preprocessing and transmission of multimodal data by calling the drivers of the camera, laser rangefinder and force sensor.

[0038] The human-machine collaboration interface is developed using Python's PyQt library, and is designed to be an intuitive, easy-to-use graphical interface for operators to monitor assembly line status, intervene in the production process, and view data information.

[0039] In this way, during the welding process of the car steering wheel frame, the visual camera continuously captures images of the welding part at a speed of 60 frames per second. The size of each frame is 2560×1920 pixels, and the collected image data is transmitted to the data acquisition unit in real time. The laser rangefinder also measures the distance between the welded parts at a frequency of 100Hz, with a measurement accuracy of ±0.05mm, and sends the distance data to the data acquisition unit synchronously. During the welding operation, the force sensor collects the force data applied by the welding gun at a sampling rate of 1000Hz with a resolution of 0.01N to ensure accurate recording of force changes.

[0040] The data acquisition unit adds an accurate timestamp to each collected data point, with a timestamp accuracy of milliseconds, to achieve accurate synchronization of different modal data in time. At the same time, a preliminary integrity check is performed on the collected data to eliminate obviously abnormal data points (such as data outside the normal measurement range).

[0041] S10, data preprocessing: Build a stable and reliable data acquisition unit to ensure that all sensors can achieve high-speed and synchronous data connection with it, and ensure the real-time and accuracy of data acquisition. Immediately pre-process the collected data in real time, use advanced filtering algorithms to remove noise interference, use statistical methods to identify and process abnormal values, and accurately mark different modal data through timestamps to ensure perfect synchronization of data in time and space, laying a solid foundation for subsequent fusion processing.

[0042] In this embodiment, the visual image data is first grayed out to convert the color image into a grayscale image, reducing the amount of data while retaining the key information of the image. Then, the median filter algorithm is used to remove the salt and pepper noise in the image, and the filter window size is set to 3×3 pixels. The image is normalized to map the pixel values ​​to a specific interval (such as [0,1]) to improve the efficiency and accuracy of subsequent deep learning model processing.

[0043] The laser distance measurement data was smoothed, and the moving average method was used to smooth the continuous distance measurement values. The window length was set to 10 to reduce the impact of measurement noise. The data collected by the force sensor was filtered, and a low-pass filter was used to remove high-frequency noise. The cutoff frequency was set to 100 Hz to ensure the stability and reliability of the force data.

[0044] The preprocessed visual image data, laser ranging data and force sensor data are fused through feature-level fusion method. Specifically, the visual image data is extracted through convolutional neural network to obtain a 1000-dimensional feature vector; the laser ranging data is processed through recurrent neural network to generate a 200-dimensional feature vector; the force sensor data is analyzed through fully connected neural network to obtain a 100-dimensional feature vector. These three feature vectors are spliced ​​to obtain a 1300-dimensional comprehensive feature vector for subsequent assembly quality inspection.

[0045] S20, Multimodal Deep Learning Model Training: An innovative multi-branch deep learning model is constructed, in which the visual image branch adopts a multi-layer convolutional neural network structure, including carefully designed convolutional layers, pooling layers and fully connected layers, to fully extract the rich texture, shape and structural features in the image; the laser ranging data branch uses a recurrent neural network structure to effectively process the timing characteristics of distance data and capture its dynamic changes; the force sensor data branch uses a multi-layer fully connected neural network to deeply analyze the characteristics of force size, direction and change trend.

[0046] In this embodiment, the specific steps of model construction and training are as follows: A multimodal deep learning model with three branches was constructed. The visual image branch used a 10-layer convolutional neural network, including 3 convolutional layers, 3 pooling layers, and 4 fully connected layers. The convolution kernel sizes of the convolutional layers were 3×3, 5×5, and 7×7, respectively, with a step size of 1, and the activation function was ReLU. The pooling layer used maximum pooling with a window size of 2×2 and a step size of 2. The number of neurons in the fully connected layers was 512, 256, 128, and 1000, respectively.

[0047] The laser ranging data branch uses a recurrent neural network containing 2 layers of LSTM (long short-term memory network) units. The number of hidden layer neurons is 100. The input sequence length is determined according to the actual length of the collected distance data sequence, and the output is a 200-dimensional feature vector.

[0048] The force sensor data branch passes through a neural network containing 3 fully connected layers with 64, 32 and 100 neurons respectively, and the activation function is ReLU.

[0049] S30, Feature Fusion: In the fusion layer of the model, advanced feature fusion strategies such as weighted summation and splicing fusion are used to efficiently integrate feature vectors from different branches, and then multiple fully connected layers are used to perform the final assembly quality classification or regression prediction task. The model is deeply trained using massive sample data covering all normal and defective assembly conditions. During the training process, the back propagation algorithm is flexibly used to fine-tune the model parameters, and combined with data enhancement techniques such as random cropping, rotation, flipping images, adding noise and other operations to expand the training samples, effectively improving the generalization ability of the model, ensuring that the model can accurately identify various assembly defects in complex actual production environments and achieve high-reliability quality inspection.

[0050] In this embodiment, in the fusion layer of the model, the feature vectors output by the three branches are concatenated, and then the assembly quality is classified by a classifier containing two fully connected layers. The number of neurons in the fully connected layer is 512 and 2 (representing the qualified and unqualified categories respectively), the activation function is Sigmoid, the loss function uses binary cross entropy loss, the optimization algorithm uses Adam optimizer, and the learning rate is initially set to 0.001.

[0051] The model was trained using 10,000 labeled sample data, of which 8,000 were used for training and 2,000 for validation. During the training process, validation was performed every 100 batches, and the learning rate was adjusted according to the accuracy on the validation set. When the accuracy did not improve for 5 consecutive cycles, the training was terminated early to prevent overfitting. After training, the model achieved an accuracy of 95% and a recall of 92% on the validation set.

[0052] In the actual detection process, the collected multimodal data is input into the trained deep learning model, and the model evaluates the assembly quality of each steering wheel frame in real time. For a sample steering wheel frame, the model outputs a probability of 0.98 that it is qualified, and it is judged as a qualified product. By further analyzing the output feature map of the middle layer of the model, it can be found that the visual image branch has obvious extraction of the weld shape and surface flatness features of the welding part, the laser ranging data branch accurately reflects that the gap between the parts meets the standard, and the force sensor data branch shows that the force change during the welding process is stable within the normal range. It is comprehensively judged that the assembly quality of the product is qualified.

[0053] S40, Reinforcement Learning Agent Training and Optimization: Define the state space of the reinforcement learning agent, comprehensively consider the feature vector of the current multimodal data, detailed statistical information of the assembly quality of the previous several inspection cycles (such as the average qualified rate, the number and proportion distribution of various defects, the average and standard deviation of product performance parameters, etc.), and fully reflect the real-time operation status of the assembly line. Set the key control parameters that can be adjusted by the assembly line as the action space of the agent, such as the precise adjustment range of the current, voltage, and welding time of the welding equipment, and the speed change range of the conveying equipment.

[0054] Carefully design the reward function and give the agent corresponding rewards or penalties according to the actual situation of the assembly quality evaluation indicators. For example, when the pass rate is significantly improved, the number of defects is greatly reduced, and the product performance parameters meet or exceed the standard requirements, the agent is given a high positive reward; conversely, when the quality decreases and the number of defects increases, a corresponding negative penalty is given. Use advanced reinforcement learning algorithms, such as deep Q-network (DQN) and proximal policy optimization (PPO), to train the agent. Through a large number of simulation experiments and online learning in actual production environments, the agent gradually masters the ability to select the optimal control parameter combination according to different assembly line states, and realizes efficient optimization control of the production process.

[0055] During the training process, simulation tests are regularly conducted on the actual production line to closely observe the actual impact of the agent's decision-making on the performance of the assembly line, including product quality fluctuations, changes in production efficiency, equipment energy consumption, etc. Based on in-depth analysis of the test results, fine-tuning and optimization of the reward function and algorithm parameters are carried out to continuously improve the accuracy, stability and adaptability of the agent's decision-making, ensuring that it can always make the best decision in a complex and changing production environment.

[0056] In this embodiment, the state space of the reinforcement learning agent is defined as a vector containing 20 elements, including the first 10 principal components of the 1300-dimensional feature vector after the current multimodal data is fused (obtained through principal component analysis), as well as the average pass rate of the first 5 inspection cycles, the number ratio of various defects in unqualified products (such as welding defects, surface scratches, etc.), and other information.

[0057] The action space of the agent includes welding current (adjustment range is 80-120A, step size is 1A), welding voltage (adjustment range is 20-30V, step size is 0.5V), welding time (adjustment range is 0.5-1.5s, step size is 0.1s) and conveying speed (adjustment range is 0.5-1.5m / s, step size is 0.1m / s).

[0058] The reward function is designed as follows: when the product qualification rate increases by 1%, the agent is given a reward of +10; when the number of defects decreases by 1, a reward of +5 is given; when the product performance parameters (such as welding strength, frame stability, etc., obtained through offline testing) increase by 5%, a reward of +20 is given. On the contrary, when the qualification rate decreases, the number of defects increases, or the performance parameters decrease, a corresponding negative penalty is given.

[0059] The proximal policy optimization (PPO) algorithm was used to train the agent, and the hyperparameters were set during the training process: batch size was 64, number of iterations was 1000, discount factor was 0.99, and entropy coefficient was 0.01. Through 10,000 trials in a simulated environment and 500 online learnings on the actual production line, the agent gradually mastered the ability to select the optimal control parameter combination according to different pipeline states.

[0060] In the actual production process, when the number of welding defects in a batch of steering wheel frames was detected to have increased slightly, the reinforcement learning agent decided to increase the welding current by 2A and extend the welding time by 0.2 seconds to improve the welding quality based on the current state of the assembly line (including multimodal data features and quality statistics of the previous cycles). At the same time, based on the changes in force sensor data during the conveying process, the agent determined that the conveying speed might be too fast, causing the frame to shake slightly, and reduced the conveying speed by 0.1 m / s to ensure the stability of the production process. After the optimization and adjustment of the agent, the qualification rate of subsequent batches of products was significantly improved from the original 90% to 95%.

[0061] S50, real-time detection: During normal operation of the assembly line, the multimodal data acquisition system continuously collects data in real time, and after efficient preprocessing, it is quickly input into the multimodal deep learning model for assembly quality inspection. The model quickly outputs the inspection results, including accurate pass / fail judgments and detailed information on defect type, location and degree.

[0062] S60, Adaptive Control: The test results and related data are transmitted to the reinforcement learning agent in real time. The agent selects the optimal control parameter adjustment plan through its training and optimized decision strategy based on the current assembly line status and reward feedback information. The assembly line control system quickly receives the decision instructions of the agent and immediately makes precise adjustments to the equipment parameters of each station, such as precise adjustment of welding current, voltage, time, conveying speed and other parameters. At the same time, the sorting mechanism quickly and accurately sorts unqualified products into the waste box based on the test results, ensuring that qualified products smoothly enter the subsequent process, achieving seamless connection and efficient operation of the production process.

[0063] In this embodiment, the deep learning model detects a steering wheel frame being produced on an assembly line and determines that it is a defective product. The main reason for the failure is the presence of porosity defects in the welding part and slight scratches on the surface of the frame. The detection system immediately feeds back the result to the assembly line control system, and stores the multimodal data (including images during welding, laser ranging data, and force sensor data), the detection results, and the unique identification information of the product in the database for subsequent quality traceability and model optimization.

[0064] After receiving the test results, the assembly line control system immediately triggers the sorting mechanism to sort the unqualified products into the waste box. At the same time, according to the decision suggestions of the reinforcement learning agent, the equipment parameters of the welding station are adjusted. The welding current is increased by 3A and the welding voltage is increased by 1V to reduce the occurrence of pore defects; the parameters of the surface treatment equipment are fine-tuned to reduce the grinding intensity to avoid further scratching the frame surface. During the production process after the parameters are adjusted, the product quality is continuously monitored. Through multimodal data collection and deep learning model detection, it is found that the product quality is gradually improved and the unqualified rate is significantly reduced.

[0065] S70, model update and continuous improvement: New sample data is collected regularly from the production line, including various normal and newly emerged defect samples, to perform incremental training on the multimodal deep learning model. By introducing new sample data, the model can continuously learn new assembly features and defect patterns, thereby updating model parameters, effectively adapting to product design changes, process improvements or newly emerged complex defect types, and always maintaining the high detection accuracy and adaptability of the model.

[0066] As the production process continues to advance, the reinforcement learning agent continues to optimize its control strategy as it accumulates experience. Through in-depth analysis of the agent's decision-making process, pipeline performance data, and comparison with actual production conditions, potential room for improvement can be discovered in a timely manner. Further fine-tuning of the reward function and algorithm parameters, introduction of new optimization algorithms or strategy improvement measures, and continuous improvement of the agent's adaptability and optimization control capabilities to complex production environments can be achieved, thereby achieving continuous optimization and upgrading of the production process.

[0067] In this embodiment, new sample data is collected from the production line regularly (weekly), including 200 normal samples and 50 samples containing new defect types (such as new welding cracks). These new samples are added to the original training data set, and the multimodal deep learning model is incrementally trained. During the training process, the transfer learning technology is used to freeze the first few convolutional layers of the model (these layers mainly learn common image and data features), and only the subsequent fully connected layers and classifier layers are retrained, and the learning rate is adjusted to 0.0001. After updated training, the model's recognition accuracy for new defect types has increased from the original 50% to 80%, and the detection accuracy of the overall assembly quality has also increased to 96%.

[0068] By analyzing the decision records and pipeline performance data of the reinforcement learning agent in the actual production process, it is found that when the product model is switched, the initial decision adaptability of the agent is poor, resulting in a temporary drop in production efficiency. To address this problem, the reward function is optimized and a reward item for adaptability to product model switching is added. When the agent can quickly adjust the control parameters after the model switch so that the production process can be restored to stability in a short time, an additional reward is given. At the same time, an exploration strategy adjustment mechanism is introduced to appropriately increase the exploration rate of the agent in the early stage of model switching to encourage it to try new control strategies. After optimization, the performance of the agent when switching product models is significantly improved, and the decline in production efficiency is reduced from the original 20% to 5%.

[0069] In this embodiment, during the system integration process, the data communication between the multimodal data acquisition system, the deep learning detection model, the reinforcement learning agent and the assembly line control system can also be fully tested. Use network monitoring tools to check indicators such as data transmission delay and packet loss rate. Under normal production load, the average data transmission delay does not exceed 10 milliseconds, and the packet loss rate is less than 0.1%, ensuring the real-time and integrity of the data. The components communicate through industrial Ethernet using the TCP / IP protocol, establish a stable and reliable communication connection, and realize efficient data interaction.

[0070] Embodiment 2 Based on the same concept, the present invention also proposes a detection device for a steering wheel frame detection line, comprising: The acquisition module is used to obtain multi-angle welding part images of the steering wheel frame of the welding station, the distance data between the welding parts, and the force change data during the welding operation, and comprehensively obtain multi-modal data; A preprocessing module, used for preprocessing multimodal data; A feature extraction module is used to extract features from the preprocessed multimodal data based on a multi-branch deep learning model, including a visual image branch, a laser ranging data branch, and a force sensor data branch; the texture, shape, and structural features in the image of the multimodal data are extracted through the visual image branch; the time series change features of the distance data of the multimodal data are extracted through the laser ranging data branch; and the size and change trend features of the force change data of the multimodal data are extracted through the force sensor data branch; The feature fusion module is used to fuse the features extracted from each branch in the fusion layer of the multi-branch deep learning model, and then perform the final assembly quality classification or regression prediction through a series of fully connected layers; the multi-branch deep learning model is deeply trained using sample data covering various normal and defective assembly situations to obtain a detection model that can be prepared to identify various types of assembly defects; The agent module is used to build a reinforcement learning agent. The key control parameters of the assembly line are set as the action space of the agent. The assembly quality evaluation index is used as the basis for the reward signal. The key control parameters are the control parameters of each device on the assembly line. The agent selects the optimal control parameter combination through reinforcement learning algorithm based on the comprehensive state of the current assembly line to maximize long-term rewards; the comprehensive state of the current assembly line is jointly characterized by multimodal data and detection results; the evaluation results of assembly quality are immediately fed back to the assembly line control system; The assembly line control system is used to store multimodal data, test results and related production information in the database; based on the test results and the decision instructions of the reinforcement learning agent, the production process is adaptively adjusted.

[0071] Preferably, a human-machine collaboration interface is also included, so that the operator can monitor the running status of the assembly line in real time through the human-machine collaboration interface. The interface displays the real-time image of the welding station, the laser ranging data curve, the force sensor data change diagram, and the position information of the product on the assembly line. When an unqualified product is detected, an alarm window pops up immediately on the interface to display the detailed defect information and location of the product. The operator can click on the alarm window to view more detailed multimodal data and test results, and judge whether manual intervention in the production process is required based on experience. For example, the operator finds that although a product is judged to be qualified, the image shows that there are some potential risks in the welding part, so the assembly line is manually paused to further inspect the product. The operator's intervention operation and feedback information are recorded and sent to the data processing unit for subsequent model optimization and intelligent agent training. After a period of human-machine collaboration, the operator's trust in the system gradually increases, and the system can also be continuously improved according to the operator's feedback, further improving the stability of the production process and product quality.

[0072] Embodiment 3 This embodiment also provides an electronic device, referring to Figure 2 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0073] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0074] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0075] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0076] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the detection methods for the steering wheel frame detection pipeline in the above embodiments.

[0077] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0078] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0079] Input / output devices 408 are used to input or output information.

[0080] Embodiment 4 This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process. The process includes the detection method for a steering wheel frame detection pipeline according to the first embodiment.

[0081] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0082] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0083] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 1 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0084] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A detection method for a steering wheel frame detection line, characterized in that: The following steps are involved: S00, acquiring multi-angle welding position images of the steering wheel frame of the welding station, distance data between welding parts and force change data during welding operation, and comprehensively obtaining multimodal data; S10, preprocessing the multimodal data; S20, performing feature extraction on the preprocessed multimodal data based on a multi-branch deep learning model, including a visual image branch, a laser ranging data branch, and a force sensor data branch; Extracting texture, shape and structural features in the image of the multimodal data through a visual image branch; Extracting time series variation characteristics of distance data of the multimodal data through a laser ranging data branch; Extracting the magnitude and change trend characteristics of the force change data of the multimodal data through the force sensor data branch; S30, in the fusion layer of the multi-branch deep learning model, the features extracted from each branch are fused, and then a series of fully connected layers are used to perform the final assembly quality classification or regression prediction; The multi-branch deep learning model is deeply trained using sample data covering various normal and defective assembly situations to obtain a detection model that is capable of identifying various types of assembly defects. S40, constructing a reinforcement learning agent, setting the key control parameters of the assembly line as the action space of the agent, and using the assembly quality evaluation index as the basis of the reward signal, wherein the key control parameters are the control parameters of each device on the assembly line; The agent selects the optimal control parameter combination through the reinforcement learning algorithm based on the comprehensive state of the current pipeline to maximize the long-term reward; the comprehensive state of the current pipeline is jointly represented by multimodal data and detection results; S50, immediately feeding back the evaluation results of the assembly quality to the assembly line control system, and storing the multimodal data, the test results and the relevant production information in the database; S60, the assembly line control system adaptively adjusts the production process based on the detection results and the decision instructions of the reinforcement learning agent.

2. A detection method for a steering wheel frame detection line according to claim 1, characterized in that: In step S10, the preprocessing includes: A filtering algorithm is used to remove noise interference, a statistical method is used to identify and process outliers, and different modal data are accurately marked by timestamps to ensure perfect synchronization of the multimodal data in time and space.

3. A detection method for a steering wheel frame detection line as claimed in claim 1, characterized in that: In step S20, the visual image branch adopts a multi-layer convolutional neural network structure; The laser ranging data branch adopts a recurrent neural network structure; The force sensor data branch adopts a multi-layer fully connected neural network.

4. A detection method for a steering wheel frame detection line as claimed in claim 1, characterized in that: In step S30, feature vectors from different branches are integrated by weighted summation or concatenation and fusion.

5. A detection method for a steering wheel frame detection line as claimed in claim 1, characterized in that: In step S30, the back-propagation algorithm is used to fine-tune the model parameters, and the training samples are expanded in combination with data enhancement technology to improve the generalization ability of the model.

6. A detection method for a steering wheel frame detection line as claimed in claim 1, characterized in that: In step S40, when constructing a reinforcement learning agent, before setting the action space of the agent, the state space of the agent is first defined. The state space is based on the feature vector after the current multimodal data is fused and the detailed statistical information of the assembly quality of the previous several detection cycles.

7. A detection method for a steering wheel frame detection line according to any one of claims 1 to 6, characterized in that: The method further includes step S70, regularly collecting new sample data from the production line, including various normal and newly appeared defect samples, and performing incremental training on the detection model.

8. A detection device for a steering wheel frame detection line, characterized in that: include: The acquisition module is used to obtain multi-angle welding part images of the steering wheel frame of the welding station, the distance data between the welding parts, and the force change data during the welding operation, and comprehensively obtain multi-modal data; A preprocessing module, used for preprocessing multimodal data; A feature extraction module is used to extract features from the preprocessed multimodal data based on a multi-branch deep learning model, including a visual image branch, a laser ranging data branch, and a force sensor data branch; The texture, shape and structural features in the image of the multimodal data are extracted through the visual image branch; the time series variation features of the distance data of the multimodal data are extracted through the laser ranging data branch; Extract the size and change trend characteristics of the force change data of the multi-modal data through the force sensor data branch; The feature fusion module is used to fuse the features extracted from each branch in the fusion layer of the multi-branch deep learning model, and then perform the final assembly quality classification or regression prediction through a series of fully connected layers; The multi-branch deep learning model is deeply trained using sample data covering various normal and defective assembly situations to obtain a detection model that is capable of identifying various types of assembly defects. The agent module is used to build a reinforcement learning agent. The key control parameters of the assembly line are set as the action space of the agent. The assembly quality evaluation index is used as the basis for the reward signal. The key control parameters are the control parameters of each device on the assembly line. The agent selects the optimal control parameter combination through reinforcement learning algorithm based on the comprehensive state of the current assembly line to maximize long-term rewards; the comprehensive state of the current assembly line is jointly characterized by multimodal data and detection results; the evaluation results of assembly quality are immediately fed back to the assembly line control system; The assembly line control system is used to store multimodal data, test results and related production information in the database; based on the test results and the decision instructions of the reinforcement learning agent, the production process is adaptively adjusted.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the detection method for a steering wheel frame detection pipeline according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes a detection method for a steering wheel frame detection pipeline according to any one of claims 1 to 7.

Images