Screw flaw detection and feeding control method and system based on machine vision

Through the multi-modal deep learning model and adaptive path planning algorithm of multi-angle image input, combined with edge computing optimization, the problems of insufficient adaptability and high error detection rate of screw defect detection and loading control in the prior art are solved, and the problem of low-level adaptability and high error detection rate is achieved. Efficient and precise screw detection and loading control are suitable for complex industrial assembly scenarios.

CN119016362APending Publication Date: 2024-11-26SUZHOU PERFECTER AUTOMATION TECH CO LTD

Patent Information

Application Number
CN202411516211.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing machine vision screw defect detection system is not adaptable when dealing with screws of different specifications and materials, has a high error detection rate, and is not highly integrated with loading control, making it difficult to meet the production needs of high efficiency and high precision.

Method used

Using machine vision-based screw defect detection and loading control methods, the pre-trained multimodal deep learning model is input to identify defect types and locations through multi-angle image input, combined with geometric dimension information for quality evaluation, and using adaptive path planning algorithm and Adaptive control strategies are accurately placed and dynamically adjusted, and the model is optimized through edge computing.

Benefits of technology

It improves the accuracy of screw detection and the accuracy of loading control, achieves high-efficiency production, enhances the robustness and adaptability of the system, can deal with various uncertainties, and is suitable for complex industrial assembly scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119016362A_ABST
    Figure CN119016362A_ABST
Patent Text Reader

Abstract

The invention provides a screw defect detection and feeding control method and system based on machine vision, and relates to the technical field of machine vision, and the method comprises the steps: obtaining a multi-angle image of a screw through a machine vision system, and inputting the multi-angle image into a pre-trained multi-mode deep learning model; according to the identified flaw type, position and geometric dimension information, in combination with a preset quality standard, evaluating the quality of the screw to obtain the quality grade of the screw; a mechanical arm is used for grabbing qualified screws and placing the qualified screws on a feeding conveying belt, based on the real-time positions and posture information of the screws, a robot is controlled to accurately place the screws to assembly positions through a self-adaptive path planning algorithm, and meanwhile obstacle avoidance and energy consumption optimization are considered; in the screw assembling process, force control parameters and movement tracks of the robot are dynamically adjusted by adopting a self-adaptive control strategy according to the material, the size and the assembling surface characteristics of the screw.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to machine vision technology, and in particular to a screw defect detection and feeding control method and system based on machine vision. Background Art

[0002] Screws are commonly used fasteners in various mechanical equipment and electronic products, and their quality directly affects the performance and reliability of the products. Traditional screw defect detection mainly relies on manual visual inspection, which has problems such as low efficiency, easy fatigue, and inconsistent judgment standards. With the development of industrial automation and intelligent manufacturing, screw defect detection technology based on machine vision has gradually emerged. However, the existing machine vision inspection system has problems such as insufficient adaptability and high false detection rate when processing screws of different specifications and materials, and the integration with feeding control is not high, which makes it difficult to meet the needs of high-efficiency and high-precision production. Summary of the invention

[0003] The embodiments of the present invention provide a screw defect detection and feeding control method and system based on machine vision, which can solve the problems in the prior art.

[0004] According to a first aspect of the embodiments of the present invention, Provides screw defect detection and feeding control methods based on machine vision, including: The multi-angle images of the screws are acquired through a machine vision system, and the multi-angle images are input into a pre-trained multi-modal deep learning model to identify the type and position of the defect on the screw surface, and extract the geometric size information of the screw. The machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multi-modal deep learning model integrates a convolutional neural network and a recurrent neural network; the quality of the screws is evaluated according to the identified defect type, position, and geometric size information, combined with a preset quality standard, to obtain the quality grade of the screws; based on the quality grade, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; Use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw conveying process, use a high-speed camera system to dynamically track the screw posture, combine the Kalman filter algorithm to predict the motion trajectory of the screw, and record the real-time position and posture information of each screw; based on the real-time position and posture information of the screw, use an adaptive path planning algorithm to control the robot to accurately place the screw to the assembly position, while considering obstacle avoidance and energy consumption optimization; during the screw assembly process, use an adaptive control strategy to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screw; use a machine vision system and posture estimation algorithm to evaluate the screw posture in real time, and trigger an automatic correction mechanism if an abnormal screw posture is detected; The data generated during the detection and loading process are processed and analyzed in real time through edge computing devices, and the multimodal deep learning model is continuously optimized using incremental learning algorithms to form a closed-loop control.

[0005] In an optional embodiment, The multimodal deep learning model integrates the steps of convolutional neural network and recurrent neural network: Constructing a multimodal deep learning model including a convolutional neural network branch and a recurrent neural network branch, wherein the convolutional neural network branch adopts a ResNet-50 network structure, and the recurrent neural network branch adopts a bidirectional long short-term memory network; Using the convolutional neural network branch to process the screw image at each angle to extract spatial features; arranging the spatial features in the order of shooting to form a time series; Using the recurrent neural network branch to process the constructed time series, capturing the dependencies between different angles, and extracting time series features; Designing an adaptive feature fusion module to fuse spatial features and temporal features, including using one-dimensional convolution to adjust the number of channels of the output features of the convolutional neural network branch and the recurrent neural network branch, splicing the adjusted features in the channel dimension, and applying a Squeeze-and-Excitation module to learn the dependencies between channels; Construct a spatiotemporal attention mechanism, where the spatial attention uses one-dimensional convolution and softmax operations to calculate the attention weight of each position on the two-dimensional feature map, and the temporal attention uses a fully connected layer and softmax operations to calculate the attention weights of different time steps; The attention weights obtained by the spatiotemporal attention mechanism are applied to the fused spatial features and temporal features to obtain weighted features.

[0006] In an optional embodiment, The method also includes optimizing and training a multimodal deep learning model: A feature pyramid network is constructed based on weighted features to generate a multi-scale feature map; a multi-task learning head is designed based on the multi-scale feature map, including a branch for defect detection and a branch for size measurement, wherein the defect detection branch includes a region proposal network, a RoI Align module, and a classification and regression head, and the size measurement branch includes a global average pooling layer and a fully connected layer; Design a multi-task joint loss function, including cross entropy loss for defect classification, smooth L1 loss for bounding box regression, mean square error loss and L2 regularization term for size regression, and introduce a learnable uncertainty parameter to dynamically adjust the weights of multi-task losses; The multimodal deep learning model is trained using a phased training strategy, including freezing the convolutional neural network backbone and training only the recurrent neural network and multi-task learning head, unfreezing the high layers of the convolutional neural network for fine-tuning, and unfreezing all layers for end-to-end fine-tuning; a cosine annealing learning rate strategy is applied during training, and a regularization method is used to prevent overfitting.

[0007] In an optional embodiment, Based on the real-time position and posture information of the screws, an adaptive path planning algorithm is used to control the robot to accurately place the screws at the assembly position. The steps of considering obstacle avoidance and energy consumption optimization include: The adaptive path planning algorithm adopts a hybrid algorithm based on the artificial potential field method and the improved fast expanding random tree algorithm; Implement multi-level obstacle avoidance strategies, including: using occupancy grid maps and expansion algorithms to handle static obstacles; using Kalman filters to predict the future motion trajectory of dynamic obstacles; introducing the concept of speed obstacles to handle multi-robot collaborative operation scenarios; Energy consumption optimization, including: using cubic spline interpolation to smooth the generated path; solving the time-optimal trajectory planning problem that satisfies the robot's dynamic constraints and converting it into a quadratic planning problem; establishing a robot joint motor energy consumption model that takes into account copper loss, iron loss, and mechanical loss; Construct a multi-objective optimization function, including three sub-goals: time, energy consumption, and accuracy. The time sub-goal is based on the result of time-optimal trajectory planning, the energy consumption sub-goal is calculated based on the energy consumption model of the robot joint motor, and the accuracy sub-goal considers the deviation between the real-time position and posture information of the screw and the target placement position. Solve the multi-objective optimization function, and generate the robot's screw placement instructions based on the results of the multi-objective optimization, combined with the adaptive path planning algorithm and multi-level obstacle avoidance strategy; During the placement process, the real-time position and posture information of the screws is continuously acquired, and the path planning and obstacle avoidance strategies are dynamically adjusted to achieve adaptive control.

[0008] In an optional embodiment, The steps of the adaptive path planning algorithm using a hybrid algorithm based on an artificial potential field method and an improved rapidly expanding random tree algorithm include: The global path is generated using an improved fast-expanding random tree algorithm, which includes adaptive sampling, heuristic bias and dynamic rewriting. Adaptive sampling dynamically adjusts the sampling density according to the complexity of the environment, heuristic bias introduces a goal-oriented sampling strategy to accelerate expansion to the goal, and dynamic rewriting periodically re-evaluates and optimizes the generated path. Based on the global path, an artificial potential field method with an adaptive potential field function is used to optimize the local path. The adaptive potential field function includes an attractive potential field term and a repulsive potential field term. The attractive potential field guides the robot to move toward the target position, and the repulsive potential field keeps the robot away from obstacles. A random disturbance term is introduced into the adaptive potential field function, and the amplitude of the random disturbance term decays exponentially with time. A local planning strategy with a rolling time domain is used to cope with dynamic environments. Local environment map updates, local path optimization, and global path deviation checks are performed in each control cycle. When it is detected that the deviation between the local path and the global path exceeds a preset threshold, an improved fast-expanding random tree algorithm is triggered to re-plan the global path. The optimized local path is smoothly spliced ​​with the global path to generate a complete motion trajectory.

[0009] In an optional embodiment, In the screw assembly process, the steps of dynamically adjusting the robot's force control parameters and motion trajectory using an adaptive control strategy according to the material, size, and assembly surface characteristics of the screw include: Establish a comprehensive force control model, including a thread contact force model that considers thread geometry and material properties, an axial preload force model based on screw material and size properties, and an assembly surface property model using Hertz contact theory; Based on the comprehensive force control model, the friction coefficient and the preload coefficient are estimated in real time using the recursive least square method; Based on the friction coefficient and preload coefficient, a fuzzy adaptive controller is designed, and the Mamdani reasoning method and the center of gravity method are used to defuzzify and dynamically adjust the force control parameters; a reinforcement learning algorithm based on policy gradient is introduced to optimize the force control parameters; The robot's motion trajectory is parameterized using a B-spline curve, and the motion trajectory is optimized using an improved differential evolution algorithm, wherein the improvements include an adaptive mutation strategy, local search enhancement, and parallel computing acceleration; Based on real-time force feedback, the optimized motion trajectory is fine-tuned online using an iterative learning control method. Apply the optimized force control parameters and motion trajectory to the robot control system to perform the screw assembly task; During the screw assembly process, force feedback information is continuously obtained, the comprehensive force control model parameters are updated, and real-time optimization of the force control parameters and motion trajectory is triggered.

[0010] In an optional embodiment, The machine vision system and posture estimation algorithm are used to evaluate the screw posture in real time. If an abnormal screw posture is detected, the steps to trigger the automatic correction mechanism include: A machine vision system is used to acquire screw images, and the acquired screw images are preprocessed in real time, including image segmentation based on local adaptive threshold, non-local mean denoising, illumination non-uniformity correction based on Retinex algorithm, and sub-pixel edge detection; Based on the preprocessed screw image, a deep learning multi-task network is used to estimate the screw posture, and the deep learning multi-task network includes a detection branch, a segmentation branch and a posture branch; based on the screw posture estimation, an iterative closest point algorithm is used to refine the screw posture and minimize the Euclidean distance between the screw CAD model point cloud and the actual screw point cloud; Define the judgment criteria for screw posture abnormality, including position deviation, angle deviation and thread orientation, and automatically adjust the judgment threshold through Bayesian optimization; The screw posture refinement results are compared with the judgment criteria for screw posture abnormality. When the screw posture abnormality is detected, the automatic correction mechanism is triggered, including path planning and compliance control. During the correction process, visual feedback information is continuously obtained, and the posture estimation and correction strategy are updated in real time.

[0011] According to a second aspect of the embodiments of the present invention, Provide screw defect detection and feeding control system based on machine vision, including: The first unit is used to obtain multi-angle images of screws through a machine vision system, input the multi-angle images into a pre-trained multi-modal deep learning model, identify the type and position of defects on the surface of the screws, and extract geometric size information of the screws. The machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multi-modal deep learning model integrates a convolutional neural network and a recurrent neural network; according to the identified defect type, position, and geometric size information, combined with a preset quality standard, the quality of the screws is evaluated to obtain the quality grade of the screws; based on the quality grade, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; The second unit is used to use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw conveying process, a high-speed camera system is used to dynamically track the screw posture, and the motion trajectory of the screw is predicted in combination with the Kalman filter algorithm, and the real-time position and posture information of each screw is recorded; based on the real-time position and posture information of the screw, an adaptive path planning algorithm is used to control the robot to accurately place the screw to the assembly position, while considering obstacle avoidance and energy consumption optimization; during the screw assembly process, an adaptive control strategy is used to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screw; the screw posture is evaluated in real time through a machine vision system and a posture estimation algorithm, and if an abnormal screw posture is detected, an automatic correction mechanism is triggered; The third unit is used to process and analyze the data generated during the detection and loading process in real time through edge computing equipment, and continuously optimize the multimodal deep learning model using an incremental learning algorithm to form a closed-loop control.

[0012] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0014] The design of the multimodal deep learning model adopted in this paper has the following beneficial effects: First, by integrating convolutional neural networks and recurrent neural networks, the model can simultaneously capture the spatial and temporal features of screw images and provide a more comprehensive feature representation. Second, the adaptive feature fusion module can flexibly adjust the importance of different modal features and improve the effect of feature fusion. Third, the spatiotemporal attention mechanism enables the model to focus on the most relevant spatial regions and time steps, improving the expressiveness of features and the interpretability of the model. Finally, this design makes full use of the information of multi-angle screw images, which helps to improve the accuracy of screw posture estimation and defect detection, and provides strong technical support for industrial quality control.

[0015] The combination of the adaptive path planning algorithm and the multi-level obstacle avoidance strategy adopted in this paper improves the robot's navigation ability in complex dynamic environments and effectively avoids collisions with static and dynamic obstacles. Secondly, the introduction of energy consumption optimization and multi-objective optimization achieves a balance between time efficiency, energy efficiency and placement accuracy, and improves the overall performance of the system. Thirdly, the acquisition of real-time position and posture information and the dynamic adjustment mechanism enhance the robustness and adaptability of the system and enable it to cope with various uncertainties. Finally, the comprehensiveness and adaptability of the solution enable it to be widely used in various complex industrial assembly scenarios to improve production efficiency and product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a method for screw defect detection and feeding control based on machine vision according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a screw defect detection and feeding control system based on machine vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0019] Figure 1 FIG. 1 is a flow chart of a method for screw defect detection and feeding control based on machine vision according to an embodiment of the present invention. Figure 1 As shown, the method includes: S1. Obtain multi-angle images of the screws through a machine vision system, input the multi-angle images into a pre-trained multimodal deep learning model, identify the type and position of defects on the surface of the screws, and extract geometric dimension information of the screws, wherein the machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multimodal deep learning model integrates a convolutional neural network and a recurrent neural network; the quality of the screws is evaluated based on the identified defect type, position, and geometric dimension information, combined with a preset quality standard, to obtain the quality grade of the screws; based on the quality grade, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; S2. Use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw conveying process, use a high-speed camera system to dynamically track the screw posture, combine the Kalman filter algorithm to predict the motion trajectory of the screw, and record the real-time position and posture information of each screw; based on the real-time position and posture information of the screw, use an adaptive path planning algorithm to control the robot to accurately place the screw to the assembly position, while considering obstacle avoidance and energy consumption optimization; during the screw assembly process, use an adaptive control strategy to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screw; use a machine vision system and posture estimation algorithm to evaluate the screw posture in real time, and trigger an automatic correction mechanism if an abnormal screw posture is detected; S3. The data generated during the detection and loading process are processed and analyzed in real time through edge computing devices, and the multimodal deep learning model is continuously optimized using incremental learning algorithms to form a closed-loop control.

[0020] In an optional embodiment, The multimodal deep learning model integrates the steps of convolutional neural network and recurrent neural network: Constructing a multimodal deep learning model including a convolutional neural network branch and a recurrent neural network branch, wherein the convolutional neural network branch adopts a ResNet-50 network structure, and the recurrent neural network branch adopts a bidirectional long short-term memory network; Using the convolutional neural network branch to process the screw image at each angle to extract spatial features; arranging the spatial features in the order of shooting to form a time series; Using the recurrent neural network branch to process the constructed time series, capturing the dependencies between different angles, and extracting time series features; Designing an adaptive feature fusion module to fuse spatial features and temporal features, including using one-dimensional convolution to adjust the number of channels of the output features of the convolutional neural network branch and the recurrent neural network branch, splicing the adjusted features in the channel dimension, and applying a Squeeze-and-Excitation module to learn the dependencies between channels; Construct a spatiotemporal attention mechanism, where the spatial attention uses one-dimensional convolution and softmax operations to calculate the attention weight of each position on the two-dimensional feature map, and the temporal attention uses a fully connected layer and softmax operations to calculate the attention weights of different time steps; The attention weights obtained by the spatiotemporal attention mechanism are applied to the fused spatial features and temporal features to obtain weighted features.

[0021] Exemplarily, the construction of a multimodal deep learning model is a complex and delicate process that requires comprehensive consideration of the extraction and fusion of spatial and temporal features. The model integrates convolutional neural networks and recurrent neural networks to fully utilize the multi-angle information of screw images.

[0022] First, a multimodal deep learning model consisting of a convolutional neural network branch and a recurrent neural network branch is constructed. The convolutional neural network branch adopts the ResNet-50 network structure, which consists of 5 stages, each of which consists of multiple residual blocks. The residual block effectively alleviates the gradient vanishing problem of the deep network through short-circuit connections, allowing the network to extract image features more deeply. The recurrent neural network branch adopts a bidirectional long short-term memory network, which includes two LSTM (long short-term memory network) layers, forward and backward, and each layer contains 128 hidden units. The bidirectional structure can consider past and future contextual information at the same time, improving the effect of temporal feature extraction.

[0023] The convolutional neural network branch processes the screw image at each angle and extracts spatial features. Specifically, the input screw image first passes through a 7x7 convolution layer and a maximum pooling layer, and then passes through the five stages of ResNet-50 in sequence. The output feature map size of each stage gradually decreases and the number of channels gradually increases. For example, for an image with an input size of 224x224x3, a 7x7x2048 feature map is obtained after processing. This feature map contains rich spatial information and can effectively characterize the shape, texture and other features of the screw.

[0024] Arrange the extracted spatial features in the order of shooting to form a time series. Assuming there are 8 screw images at different angles, the dimension of the time series is 8x2048, where 8 represents the time step and 2048 represents the feature dimension. This time series contains the spatial feature change information of the screw at different angles.

[0025] The recurrent neural network branch processes the constructed time series, captures the dependencies between different angles, and extracts time series features. The bidirectional LSTM network processes the time series from both the forward and reverse directions, and the LSTM layer in each direction outputs a 128-dimensional feature vector. The two feature vectors are concatenated to obtain a 256-dimensional time series feature, which contains the changing rules and dependencies of the screws at different angles.

[0026] An adaptive feature fusion module is designed to fuse spatial features and temporal features. First, one-dimensional convolution is used to adjust the number of channels of the output features of the convolutional neural network branch and the recurrent neural network branch. For example, the number of channels of spatial features is adjusted from 2048 to 512, and the number of channels of temporal features is adjusted from 256 to 512. Then, the adjusted features are concatenated in the channel dimension to obtain 1024-channel fused features. Next, the Squeeze-and-Excitation module is applied to learn the dependencies between channels. This module first performs global average pooling on the features, then generates channel weights through two fully connected layers and a sigmoid activation function, and finally applies the weights to the original features to obtain weighted fused features.

[0027] Construct a spatiotemporal attention mechanism, including spatial attention and temporal attention. Spatial attention uses one-dimensional convolution and softmax operations to calculate the attention weights of each position on the two-dimensional feature map. Specifically, a 1x1 convolution is applied to the fused features to reduce the number of channels to 1, and then the spatial attention weights are normalized by the softmax function. Temporal attention uses a fully connected layer and a softmax operation to calculate the attention weights at different time steps. First, the features are flattened, mapped to the time step dimension through a fully connected layer, and then normalized by a softmax function to obtain the temporal attention weights.

[0028] The attention weights obtained by the spatiotemporal attention mechanism are applied to the fused spatial features and temporal features to obtain weighted features. Specifically, the spatial attention weights are multiplied by the fused features in the spatial dimension, and the temporal attention weights are multiplied by the fused features in the temporal dimension. The final weighted features retain important spatial information while highlighting key temporal changes.

[0029] The design of the multimodal deep learning model adopted in this paper has the following beneficial effects: First, by integrating convolutional neural networks and recurrent neural networks, the model can simultaneously capture the spatial and temporal features of screw images and provide a more comprehensive feature representation. Second, the adaptive feature fusion module can flexibly adjust the importance of different modal features and improve the effect of feature fusion. Third, the spatiotemporal attention mechanism enables the model to focus on the most relevant spatial regions and time steps, improving the expressiveness of features and the interpretability of the model. Finally, this design makes full use of the information of multi-angle screw images, which helps to improve the accuracy of screw posture estimation and defect detection, and provides strong technical support for industrial quality control.

[0030] In an optional embodiment, The method also includes optimizing and training a multimodal deep learning model: A feature pyramid network is constructed based on weighted features to generate a multi-scale feature map; a multi-task learning head is designed based on the multi-scale feature map, including a branch for defect detection and a branch for size measurement, wherein the defect detection branch includes a region proposal network, a RoI Align module, and a classification and regression head, and the size measurement branch includes a global average pooling layer and a fully connected layer; Design a multi-task joint loss function, including cross entropy loss for defect classification, smooth L1 loss for bounding box regression, mean square error loss and L2 regularization term for size regression, and introduce a learnable uncertainty parameter to dynamically adjust the weights of multi-task losses; The multimodal deep learning model is trained using a phased training strategy, including freezing the convolutional neural network backbone and training only the recurrent neural network and multi-task learning head, unfreezing the high layers of the convolutional neural network for fine-tuning, and unfreezing all layers for end-to-end fine-tuning; a cosine annealing learning rate strategy is applied during training, and a regularization method is used to prevent overfitting.

[0031] For example, the optimization and training of multimodal deep learning models are key steps to achieve efficient screw defect detection and size measurement. The process includes multiple steps such as feature extraction, multi-task learning head design, loss function construction, and training strategy formulation.

[0032] First, a feature pyramid network is constructed based on weighted features to generate multi-scale feature maps. The feature pyramid network adopts a top-down structure to gradually fuse low-level fine-grained features from high-level semantic features. Specifically, based on the 5-stage output of ResNet-50, feature maps of 5 scale levels are constructed. The number of channels of the feature map of each scale level is adjusted to 256 through 1x1 convolution. Features are fused between adjacent scale levels through upsampling and lateral connections. For example, for an image with an input size of 1024x1024, the sizes of the generated multi-scale feature maps are 32x32, 64x64, 128x128, 256x256, and 512x512, respectively, and the number of channels is 256.

[0033] Based on the generated multi-scale feature map, a multi-task learning head is designed, including a defect detection branch and a size measurement branch. The defect detection branch first uses a region proposal network to generate candidate regions. The region proposal network sets multiple anchor boxes of different scales and aspect ratios at each feature map position, and predicts the category probability and bounding box regression parameters of the anchor box through 3x3 convolution and 1x1 convolution respectively. Then, a fixed-size feature map, such as 7x7, is extracted from the multi-scale feature map using the RoI Align module (region of interest alignment). Finally, classification and regression are implemented through a fully connected layer, and the defect category probability and precise bounding box coordinates are output. The size measurement branch uses a global average pooling layer to compress the feature map into a one-dimensional vector, and then outputs the size information such as the length and diameter of the screw through a fully connected layer.

[0034] A multi-task joint loss function is designed to comprehensively consider the optimization objectives of defect detection and size measurement tasks. Defect classification uses cross entropy loss to measure the difference between the predicted category probability and the true category. Bounding box regression uses smooth L1 loss, which has a smoother gradient near the zero point than ordinary L1 loss, which is conducive to stable training of the model. Size regression uses mean square error loss to directly measure the gap between the predicted size and the true size. At the same time, the L2 regularization term is introduced to prevent overfitting by imposing penalties on model parameters. In order to balance the importance of multiple tasks, a learnable uncertainty parameter is introduced to dynamically weight the loss of each task. Specifically, each task loss term is divided by the corresponding uncertainty parameter and the logarithm of the parameter is added. In this way, the model can adaptively adjust the relative importance of each task during training.

[0035] A phased training strategy is used to train the multimodal deep learning model to achieve better optimization results. First, the convolutional neural network backbone is frozen, and only the recurrent neural network and multi-task learning head are trained. In this stage, a larger learning rate, such as 0.01, is used for 30 iterations. Then the high-level convolutional neural network is unfrozen for fine-tuning, specifically the last two stages of ResNet-50. In this stage, a smaller learning rate, such as 0.001, is used for 20 iterations. Finally, all layers are unfrozen for end-to-end fine-tuning, using an even smaller learning rate, such as 0.0001, for 10 iterations. The cosine annealing learning rate strategy is applied throughout the training process. The learning rate decays periodically as the training progresses, which helps the model jump out of the local optimum. At the same time, multiple regularization methods are used to prevent overfitting, including weight decay (set to 0.0001), random dropout (set to 0.5), and early stopping.

[0036] This optimization and training method has the following beneficial effects: First, the design of the feature pyramid network and the multi-task learning head enables the model to simultaneously process defects of different scales and perform precise dimensional measurements, which improves the adaptability and accuracy of the model. Second, the design of the multi-task joint loss function realizes the collaborative optimization of defect detection and dimensional measurement tasks, and achieves a more flexible multi-task balance through learnable task weights. Third, the application of phased training strategies and various optimization techniques improves the training efficiency and generalization ability of the model, and effectively prevents overfitting. Finally, this optimization and training method enables the multimodal deep learning model to better adapt to complex industrial scenarios, providing an efficient and accurate technical solution for screw quality control.

[0037] In an optional embodiment, Based on the real-time position and posture information of the screws, an adaptive path planning algorithm is used to control the robot to accurately place the screws at the assembly position. The steps of considering obstacle avoidance and energy consumption optimization include: The adaptive path planning algorithm adopts a hybrid algorithm based on the artificial potential field method and the improved fast expanding random tree algorithm; Implement multi-level obstacle avoidance strategies, including: using occupancy grid maps and expansion algorithms to handle static obstacles; using Kalman filters to predict the future motion trajectory of dynamic obstacles; introducing the concept of speed obstacles to handle multi-robot collaborative operation scenarios; Energy consumption optimization, including: using cubic spline interpolation to smooth the generated path; solving the time-optimal trajectory planning problem that satisfies the robot's dynamic constraints and converting it into a quadratic planning problem; establishing a robot joint motor energy consumption model that takes into account copper loss, iron loss, and mechanical loss; Construct a multi-objective optimization function, including three sub-goals: time, energy consumption, and accuracy. The time sub-goal is based on the result of time-optimal trajectory planning, the energy consumption sub-goal is calculated based on the energy consumption model of the robot joint motor, and the accuracy sub-goal considers the deviation between the real-time position and posture information of the screw and the target placement position. Solve the multi-objective optimization function, and generate the robot's screw placement instructions based on the results of the multi-objective optimization, combined with the adaptive path planning algorithm and multi-level obstacle avoidance strategy; During the placement process, the real-time position and posture information of the screws is continuously acquired, and the path planning and obstacle avoidance strategies are dynamically adjusted to achieve adaptive control.

[0038] For example, the design of the robot control system is a key link in achieving accurate screw placement, which requires comprehensive consideration of path planning, obstacle avoidance strategy, energy consumption optimization, etc. This technical solution adopts an adaptive path planning algorithm, combined with a multi-level obstacle avoidance strategy and energy consumption optimization method, to achieve high-precision and high-efficiency screw placement.

[0039] The adaptive path planning algorithm uses a hybrid algorithm based on the artificial potential field method and the improved rapidly expanding random tree algorithm. The artificial potential field method guides the robot's motion by constructing a gravitational field and a repulsive field, where the target point generates gravitational force and the obstacles generate repulsive force. The improved rapidly expanding random tree algorithm constructs a tree structure by randomly sampling and connecting nodes in the configuration space, thereby quickly exploring complex environments. The hybrid algorithm combines the advantages of both methods, expanding the random tree under the guidance of the artificial potential field, and improving the efficiency and robustness of path planning. For example, in a 2D environment with multiple obstacles, the algorithm is able to generate a smooth path that avoids all obstacles within 0.1 seconds.

[0040] The implementation of the multi-level obstacle avoidance strategy includes three levels. First, static obstacles are handled using an occupancy grid map and an expansion algorithm. The workspace is discretized into a 50x50x50 three-dimensional grid, and each grid is marked as occupied or idle. The occupancy grid is expanded, and the expansion radius is the maximum radius of the robot plus a safety margin, such as 20cm. Second, the Kalman filter is used to predict the future motion trajectory of dynamic obstacles. Assuming that the motion of dynamic obstacles conforms to the uniform linear motion model, the state estimate is updated through continuous observation data. The prediction time window is set to 2 seconds, and the prediction results are updated every 100ms. Finally, the concept of speed obstacles is introduced to handle multi-robot collaborative operation scenarios. The motion of other robots is regarded as a speed obstacle, and a time-varying prohibited area is constructed based on their current position and speed.

[0041] The energy consumption optimization process includes three main steps. First, the generated path is smoothed using cubic spline interpolation. Control points are evenly selected on the path, and these control points are connected by cubic spline functions to obtain a continuous and differentiable smooth trajectory. Then, the time optimal trajectory planning problem that satisfies the robot dynamics constraints is solved and converted into a quadratic programming problem. Considering the position, velocity and acceleration limitations of the robot joints, the constraints are constructed. The objective function is set to minimize the total execution time. Finally, an energy consumption model of the robot joint motor considering copper loss, iron loss and mechanical loss is established. Copper loss is proportional to the square of the current, iron loss is proportional to the 1.5th power of the rotational speed, and mechanical loss is proportional to the rotational speed. The coefficients of each loss are measured experimentally, for example, the copper loss coefficient is 0.1W / A 2 , iron loss coefficient is 0.05W / (rad / s) 1.5 , the mechanical loss coefficient is 0.02W / (rad / s).

[0042] Construct a multi-objective optimization function, including three sub-goals: time, energy consumption, and accuracy. The time sub-goal is based on the result of the time-optimal trajectory planning, and the total execution time is directly used as the evaluation index. The energy consumption sub-goal is calculated based on the energy consumption model of the robot joint motor, and the energy consumption during the entire trajectory execution process is accumulated. The accuracy sub-goal considers the deviation between the real-time position and posture information of the screw and the target placement position, and uses the weighted sum of the Euclidean distance and the posture angle difference as the evaluation index. In order to balance the three sub-goals, weight coefficients are introduced, such as a time weight of 0.4, an energy weight of 0.3, and an accuracy weight of 0.3.

[0043] The non-dominated sorting genetic algorithm II (NSGA-II) is used to solve the multi-objective optimization function. The population size is set to 100 and the number of evolution generations is set to 50. Binary tournament selection, simulated binary crossover, and polynomial mutation are used as genetic operations. The diversity and convergence of the population are maintained by non-dominated sorting and crowding distance calculation. Finally, a set of Pareto optimal solutions are obtained, from which the best compromise solution is selected as the final optimization result.

[0044] Based on the results of multi-objective optimization, combined with the adaptive path planning algorithm and multi-level obstacle avoidance strategy, the robot's screw placement instructions are generated. Specifically, the optimized trajectory is discretized into a series of path points, each of which contains position and posture information. According to the robot's kinematic model, the path points are converted into motion instructions in the joint space. At the same time, considering the opening and closing state of the robot's end effector, the operation of grasping and releasing the screw is performed at the appropriate time.

[0045] During the placement process, the real-time position and posture information of the screws is continuously obtained, and the path planning and obstacle avoidance strategies are dynamically adjusted to achieve adaptive control. A high-precision visual system or force feedback sensor is used to obtain the position and posture information of the screws, with an update frequency of 100Hz. When it is detected that the screw position or posture deviation exceeds the preset threshold (for example, the position deviation is greater than 1mm or the posture deviation is greater than 1 degree), the path re-planning is triggered. During the re-planning process, the current position is used as the starting point and the original target position is used as the end point, and the aforementioned adaptive path planning algorithm is used to quickly generate a new trajectory. At the same time, the predicted trajectory of dynamic obstacles and the prohibited area of ​​speed obstacles are updated to ensure the real-time and effectiveness of the obstacle avoidance strategy.

[0046] The combination of the adaptive path planning algorithm and the multi-level obstacle avoidance strategy adopted in this paper improves the robot's navigation ability in complex dynamic environments and effectively avoids collisions with static and dynamic obstacles. Secondly, the introduction of energy consumption optimization and multi-objective optimization achieves a balance between time efficiency, energy efficiency and placement accuracy, and improves the overall performance of the system. Thirdly, the acquisition of real-time position and posture information and the dynamic adjustment mechanism enhance the robustness and adaptability of the system and enable it to cope with various uncertainties. Finally, the comprehensiveness and adaptability of the solution enable it to be widely used in various complex industrial assembly scenarios to improve production efficiency and product quality.

[0047] In an optional embodiment, The steps of the adaptive path planning algorithm using a hybrid algorithm based on an artificial potential field method and an improved rapidly expanding random tree algorithm include: The global path is generated using an improved fast-expanding random tree algorithm, which includes adaptive sampling, heuristic bias and dynamic rewriting. Adaptive sampling dynamically adjusts the sampling density according to the complexity of the environment, heuristic bias introduces a goal-oriented sampling strategy to accelerate expansion to the goal, and dynamic rewriting periodically re-evaluates and optimizes the generated path. Based on the global path, an artificial potential field method with an adaptive potential field function is used to optimize the local path. The adaptive potential field function includes an attractive potential field term and a repulsive potential field term. The attractive potential field guides the robot to move toward the target position, and the repulsive potential field keeps the robot away from obstacles. A random disturbance term is introduced into the adaptive potential field function, and the amplitude of the random disturbance term decays exponentially with time. A local planning strategy with a rolling time domain is used to cope with dynamic environments. Local environment map updates, local path optimization, and global path deviation checks are performed in each control cycle. When it is detected that the deviation between the local path and the global path exceeds a preset threshold, an improved fast-expanding random tree algorithm is triggered to re-plan the global path. The optimized local path is smoothly spliced ​​with the global path to generate a complete motion trajectory.

[0048] Exemplarily, the adaptive path planning algorithm is a hybrid algorithm combining an improved rapidly expanding random tree algorithm and an artificial potential field method, which is designed to solve the problem of robot navigation in complex dynamic environments. The algorithm achieves efficient and robust path planning through multiple key steps.

[0049] First, a global path is generated using an improved rapidly expanding random tree algorithm. This step contains three major improvements: adaptive sampling, heuristic bias, and dynamic rewriting. Adaptive sampling dynamically adjusts the sampling density according to the complexity of the environment, increasing sampling points in areas with dense obstacles and reducing sampling points in open areas. For example, when the number of obstacles detected in the local area exceeds 10, the sampling density is increased to 2 times the original; when the number of obstacles is less than 3, the sampling density is reduced to 0.5 times the original. Heuristic bias introduces a goal-oriented sampling strategy to accelerate expansion toward the target. Specifically, on the basis of random sampling, there is an 80% probability of sampling in a cone area in the target direction, and the cone angle is set to 60 degrees. The dynamic rewriting mechanism periodically re-evaluates and optimizes the generated path, performing a rewrite operation every 100 nodes or 5 seconds to remove redundant nodes and optimize local connections.

[0050] Next, based on the global path, the local path optimization is performed using the artificial potential field method with an adaptive potential field function. The adaptive potential field function includes an attractive potential field term and a repulsive potential field term. The attractive potential field guides the robot to move toward the target position, and its strength is proportional to the distance from the robot to the target, with the proportional coefficient set to 0.5. The repulsive potential field keeps the robot away from obstacles, and its strength is inversely proportional to the distance from the robot to the obstacle, with the influence radius set to 2 meters. In order to avoid the local minimum problem, a random perturbation term is introduced in the adaptive potential field function. The initial amplitude of the random perturbation term is set to 10% of the maximum value of the potential field function, and it decays exponentially over time with a decay coefficient of 0.95.

[0051] To cope with dynamic environments, a local planning strategy with a rolling time domain is adopted. The following operations are performed in each control cycle (usually 100 milliseconds): First, the local environment map is updated, and obstacle information within a range of 5 meters is obtained using a lidar or depth camera, and the grid map is updated. Then, local path optimization is performed, and the above-mentioned adaptive potential field function is used to generate the optimal local path within a range of 1 meter. Finally, a global path deviation check is performed to calculate the vertical distance between the local path end point and the global path. When it is detected that the deviation between the local path and the global path exceeds a preset threshold (such as 0.5 meters), the improved fast expansion random tree algorithm is triggered to re-plan the global path.

[0052] The last step is to smoothly splice the optimized local path with the global path to generate a complete motion trajectory. This step uses a cubic Bezier curve to achieve a smooth transition. The last point of the local path and the point closest to the point on the global path are selected as control points. The tangent direction of the curve is consistent with the tangent direction of the local path and the global path at these two points. By adjusting the weight of the control point, the smoothness of the transition can be controlled. The weight is usually set between 0.3 and 0.7.

[0053] To better understand the operation of the algorithm, consider the following specific case: Assume that in a 10m×10m workspace, the starting point coordinates are (0, 0), the target point coordinates are (9, 9), and there are multiple static obstacles and a dynamic obstacle moving at a speed of 1m / s in the middle. First, the improved fast-expanding random tree algorithm generates an initial global path in 0.5 seconds, containing about 100 path points. Then, the adaptive potential field function optimizes the local path within 1 meter in each control cycle (100 milliseconds). When a dynamic obstacle approaches, the local path will deviate. If the deviation exceeds 0.5m, global replanning is triggered, and a new global path is generated within 0.3 seconds. During the entire navigation process, the robot can smoothly avoid all obstacles and reach the target position in about 15 seconds.

[0054] The adaptive path planning algorithm adopted in this paper has the following beneficial effects: First, by combining global planning and local optimization, the algorithm can quickly generate feasible paths in complex dynamic environments with good real-time performance. Second, adaptive sampling and heuristic bias improve the efficiency of path generation, especially in obstacle-dense areas and target directions. Third, the dynamic rewriting mechanism and rolling time domain strategy enhance the algorithm's adaptability to environmental changes and can respond to newly emerging obstacles in a timely manner. Finally, the smooth splicing technology ensures the continuity and executability of the generated trajectory, which is conducive to the stable control of the robot. Overall, the algorithm achieves a good balance between path quality, computational efficiency, and environmental adaptability, providing an effective solution for robot autonomous navigation in complex scenarios.

[0055] In an optional embodiment, In the screw assembly process, the steps of dynamically adjusting the robot's force control parameters and motion trajectory using an adaptive control strategy according to the material, size, and assembly surface characteristics of the screw include: Establish a comprehensive force control model, including a thread contact force model that considers thread geometry and material properties, an axial preload force model based on screw material and size properties, and an assembly surface property model using Hertz contact theory; Based on the comprehensive force control model, the friction coefficient and the preload coefficient are estimated in real time using the recursive least square method; Based on the friction coefficient and preload coefficient, a fuzzy adaptive controller is designed, and the Mamdani reasoning method and the center of gravity method are used to defuzzify and dynamically adjust the force control parameters; a reinforcement learning algorithm based on policy gradient is introduced to optimize the force control parameters; The robot's motion trajectory is parameterized using a B-spline curve, and the motion trajectory is optimized using an improved differential evolution algorithm, wherein the improvements include an adaptive mutation strategy, local search enhancement, and parallel computing acceleration; Based on real-time force feedback, the optimized motion trajectory is fine-tuned online using an iterative learning control method. Apply the optimized force control parameters and motion trajectory to the robot control system to perform the screw assembly task; During the screw assembly process, force feedback information is continuously obtained, the comprehensive force control model parameters are updated, and real-time optimization of the force control parameters and motion trajectory is triggered.

[0056] For example, the adaptive control strategy in the screw assembly process is the key to achieving high-precision and high-efficiency assembly. This strategy dynamically adjusts the robot's force control parameters and motion trajectory to adapt to the material, size and assembly surface characteristics of different screws, thereby improving assembly quality and efficiency.

[0057] First, a comprehensive force control model is established, which includes three sub-models: thread contact force model, axial preload model and assembly surface characteristic model. The thread contact force model takes into account the geometric shape and material properties of the thread, such as pitch, thread angle, thread surface roughness, etc. For M8 screws, the pitch is usually 1.25mm, the thread angle is 60 degrees, and the surface roughness Ra value is about 3.2μm. The axial preload model is based on the material and dimensional characteristics of the screw, considering the yield strength, tensile strength and cross-sectional area of ​​the screw. For example, for 8.8 grade M8 bolts, the yield strength is about 640MPa, the tensile strength is about 800MPa, and the effective cross-sectional area is about 36.6mm². The assembly surface characteristic model adopts Hertz contact theory, considering the elastic modulus, Poisson's ratio and surface shape of the contact surface. Taking the aluminum alloy surface as an example, the elastic modulus is about 70GPa and the Poisson's ratio is about 0.33.

[0058] Based on the comprehensive force control model, the recursive least squares method is used to estimate the friction coefficient and preload coefficient in real time. The forgetting factor of the recursive least squares method is set to 0.98, and the diagonal elements of the initial covariance matrix are set to 100. The estimated value is updated every 10ms based on the real-time force sensor data and the screw rotation angle data. For example, for a well-lubricated steel-steel contact, the initial estimate of the friction coefficient can be set to 0.15, and the initial estimate of the preload coefficient can be set to 0.2.

[0059] According to the estimated friction coefficient and preload coefficient, a fuzzy adaptive controller is designed. The controller uses Mamdani reasoning method and center of gravity method to defuzzify and dynamically adjust the force control parameters. The input variables include friction coefficient error and preload coefficient error, and the output variable is the force control parameter adjustment amount. Five fuzzy sets (negative large, negative small, zero, positive small, positive large) are defined for each input variable, and seven fuzzy sets are defined for the output variable. The fuzzy rule base contains 25 rules, such as "if the friction coefficient error is positive large and the preload coefficient error is negative small, then the force control parameter adjustment amount is positive medium". At the same time, a reinforcement learning algorithm based on policy gradient is introduced to optimize the force control parameters. The learning rate is set to 0.01, the discount factor is set to 0.9, and the policy network is updated every 100 assembly iterations.

[0060] The robot's motion trajectory is parameterized using B-spline curves and optimized using an improved differential evolution algorithm. The B-spline curve uses cubic basis functions and the number of control points is 10. The improved differential evolution algorithm includes three main improvements: adaptive mutation strategy, local search enhancement, and parallel computing acceleration. The adaptive mutation strategy dynamically adjusts the scaling factor according to the population diversity. The initial value is set to 0.5, which increases as the population diversity decreases, and the maximum value is 1.0. The local search enhancement performs a local search for the optimal individual after each generation of evolution. The initial value of the search radius is 1% of the control point coordinate range and decreases with the number of iterations. Parallel computing uses the OpenMP framework and is implemented on an 8-core CPU. The population size is set to 100 and the maximum number of iterations is 1000.

[0061] Based on real-time force feedback, the optimized motion trajectory is fine-tuned online using an iterative learning control method. The P-type learning law is used, and the learning gain is set to 0.1. After each assembly is completed, the trajectory parameters are updated according to the force error sequence. For example, if the axial force at a certain time point is detected to be too large, the axial displacement near that time point is reduced.

[0062] The optimized force control parameters and motion trajectory are applied to the robot control system to perform the screw assembly task. The force control parameters include the stiffness matrix and the damping matrix. The initial value of the stiffness matrix is ​​set to a 6x6 diagonal matrix with diagonal elements of (1000, 1000, 1000, 100, 100) N / m, and the initial value of the damping matrix is ​​set to a 6x6 diagonal matrix with diagonal elements of (50, 50, 50, 5, 5, 5) Ns / m. The motion trajectory is defined by the optimized B-spline curve and converted to the joint space trajectory through inverse kinematics.

[0063] During the screw assembly process, force feedback information is continuously obtained, the comprehensive force control model parameters are updated, and real-time optimization of force control parameters and motion trajectories is triggered. Force feedback information is obtained through a six-dimensional force / torque sensor with a sampling frequency of 1kHz. When the force error is detected to exceed the preset threshold (for example, the axial force error is greater than 2N or the torque error is greater than 0.1Nm), the parameter update and optimization process is triggered. The update and optimization are performed in the background thread and do not affect the execution of the current assembly task.

[0064] The adaptive control strategy adopted in this paper has the following beneficial effects: First, the establishment of a comprehensive force control model and real-time parameter estimation enable the control system to accurately capture the mechanical characteristics of the screw assembly process, providing a reliable basis for subsequent optimization. Second, the combination of the fuzzy adaptive controller and the reinforcement learning algorithm realizes the intelligent adjustment of the force control parameters, which can adapt to the characteristic changes of different screws and assembly surfaces. Third, the trajectory optimization based on the B-spline curve and the improved differential evolution algorithm improves the smoothness and efficiency of the motion and reduces the impact and vibration during the assembly process. Finally, the iterative learning control and real-time optimization mechanism enhance the adaptive ability and robustness of the system, and can cope with various uncertainties in the assembly process. Overall, this strategy significantly improves the accuracy, efficiency and reliability of screw assembly, and provides an effective solution for high-quality automated assembly.

[0065] In an optional embodiment, The machine vision system and posture estimation algorithm are used to evaluate the screw posture in real time. If an abnormal screw posture is detected, the steps to trigger the automatic correction mechanism include: A machine vision system is used to acquire screw images, and the acquired screw images are preprocessed in real time, including image segmentation based on local adaptive threshold, non-local mean denoising, illumination non-uniformity correction based on Retinex algorithm, and sub-pixel edge detection; Based on the preprocessed screw image, a deep learning multi-task network is used to estimate the screw posture, and the deep learning multi-task network includes a detection branch, a segmentation branch and a posture branch; based on the screw posture estimation, an iterative closest point algorithm is used to refine the screw posture and minimize the Euclidean distance between the screw CAD model point cloud and the actual screw point cloud; Define the judgment criteria for screw posture abnormality, including position deviation, angle deviation and thread orientation, and automatically adjust the judgment threshold through Bayesian optimization; The screw posture refinement results are compared with the judgment criteria for screw posture abnormality. When the screw posture abnormality is detected, the automatic correction mechanism is triggered, including path planning and compliance control. During the correction process, visual feedback information is continuously obtained, and the posture estimation and correction strategy are updated in real time.

[0066] For example, real-time posture evaluation and automatic correction during screw assembly is a key technology to ensure assembly quality and efficiency. Through machine vision systems and advanced posture estimation algorithms, screw posture anomalies can be detected and corrected in a timely manner, thereby improving assembly success rate and efficiency.

[0067] First, a machine vision system is used to acquire screw images. The system includes a high-resolution industrial camera and a structured light projector. The camera resolution is 4096x3000 pixels and the frame rate is 60fps. The acquired images undergo a series of real-time preprocessing steps to improve the accuracy of subsequent pose estimation. The preprocessing steps include image segmentation based on local adaptive threshold, non-local mean denoising, illumination inhomogeneity correction based on the Retinex algorithm, and sub-pixel edge detection.

[0068] In image segmentation, the local adaptive threshold method is used, the window size is set to 1 / 16 of the image width, and the constant C is set to 5. This method can effectively process images under uneven lighting conditions. The search window size of the non-local mean denoising algorithm is set to 21x21, the similarity window size is 7x7, and the filter strength h is set to 10. The illumination unevenness correction based on the Retinex algorithm (retina perception algorithm) adopts the multi-scale Retinex algorithm, the number of scales is set to 3, and the weights are 0.3, 0.4 and 0.3 respectively. This step can effectively reduce the impact of shadows and highlights on subsequent processing. The improved Canny algorithm is used for sub-pixel edge detection, the low threshold is set to 50, the high threshold is set to 150, and the σ value is set to 1.4. This method can provide more accurate screw contour information.

[0069] Based on the preprocessed screw images, a deep learning multi-task network is used to estimate the screw pose. The network includes detection branch, segmentation branch and pose branch. The detection branch adopts a single-stage target detection network architecture to locate the position of the screw in the image. The segmentation branch adopts the U-Net architecture to generate an accurate contour mask of the screw. The pose branch uses a residual network as the backbone network to output 6-DOF pose parameters. The network training adopts a hybrid loss function, including detection loss, segmentation loss and pose loss, with weights of 0.3, 0.3 and 0.4 respectively. The training dataset contains 10,000 annotated screw images, 80% of which are used for training and 20% for validation. The adaptive moment estimation optimizer is used in the training process, and the initial learning rate is set to 0.001, which is decayed by 10% every 50 training cycles.

[0070] To further improve the accuracy of pose estimation, the iterative closest point (ICP) algorithm is used to refine the screw pose. First, the estimated pose parameters are applied to the CAD model of the screw to generate an initial point cloud. Then, the Euclidean distance between the CAD model point cloud and the actual screw point cloud is minimized through iterative optimization using the ICP algorithm. The maximum number of iterations of the ICP algorithm is set to 50, and the convergence threshold is set to 0.001 mm. Through this refinement process, the average error of pose estimation can be reduced from the initial 2 mm and 2° to 0.5 mm and 0.5°.

[0071] Define the judgment criteria for screw posture anomalies, including position deviation, angle deviation, and thread orientation. The initial thresholds are set as follows: position deviation does not exceed 2 mm, angle deviation does not exceed 3°, and thread orientation deviation does not exceed 5°. In order to adapt to different working conditions and screw types, Bayesian optimization is used to automatically adjust these judgment thresholds. During the Bayesian optimization process, the Gaussian process regression model is used, the acquisition function is the expected improvement (EI), the initial number of sampling points is 20, and the maximum number of iterations is 100. The optimization goal is to minimize the weighted sum of the false positive rate and the missed positive rate, with weights of 0.6 and 0.4, respectively.

[0072] When an abnormal screw posture is detected, the automatic correction mechanism is triggered. The mechanism consists of two main parts: path planning and compliance control. The path planning adopts the improved Rapidly Expanding Random Tree (RRT*) algorithm, taking into account the obstacles and kinematic constraints of the robot workspace. The sampling number of the RRT* algorithm is set to 10,000 and the rewiring radius is set to 20 mm. The generated path is smoothed by cubic spline interpolation to ensure the continuity of the motion.

[0073] The compliant control adopts the impedance control strategy to dynamically adjust the stiffness and damping parameters of the robot end effector. The initial value of the stiffness matrix is ​​set to a 6x6 diagonal matrix with diagonal elements of (1000, 1000, 1000, 100, 100) N / m, and the initial value of the damping matrix is ​​set to a 6x6 diagonal matrix with diagonal elements of (50, 50, 50, 5, 5, 5) Ns / m. During the correction process, these parameters are adjusted in real time according to the change of contact force to achieve compliant posture correction.

[0074] During the correction process, visual feedback information is continuously obtained to update the posture estimation and correction strategy in real time. Visual feedback uses 60fps high-speed image acquisition and GPU acceleration to achieve real-time processing. The posture estimation update frequency is 30Hz, and the correction strategy update frequency is 100Hz. When it is detected that the posture has met the requirements, the correction process is automatically terminated and the next assembly operation is entered.

[0075] This paper achieves fast and accurate estimation of the screw posture through efficient image preprocessing and deep learning multi-task network, providing a reliable basis for subsequent correction. Secondly, the ICP algorithm is used for posture refinement, which further improves the accuracy of posture estimation and makes the correction process more accurate. Thirdly, the adaptive threshold adjustment mechanism based on Bayesian optimization enhances the system's adaptability to different working conditions. Finally, the automatic correction mechanism combining RRT* path planning and compliant control is used to achieve efficient and safe posture correction. Overall, this solution significantly improves the robustness and efficiency of the screw assembly process, reduces the assembly failure rate and product damage risk, and provides strong support for high-quality automated assembly.

[0076] Figure 2 FIG. 1 is a schematic diagram of a screw defect detection and feeding control system based on machine vision according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to obtain multi-angle images of screws through a machine vision system, input the multi-angle images into a pre-trained multi-modal deep learning model, identify the type and position of defects on the surface of the screws, and extract geometric size information of the screws. The machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multi-modal deep learning model integrates a convolutional neural network and a recurrent neural network; according to the identified defect type, position, and geometric size information, combined with a preset quality standard, the quality of the screws is evaluated to obtain the quality grade of the screws; based on the quality grade, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; The second unit is used to use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw conveying process, a high-speed camera system is used to dynamically track the screw posture, and the motion trajectory of the screw is predicted in combination with the Kalman filter algorithm, and the real-time position and posture information of each screw is recorded; based on the real-time position and posture information of the screw, an adaptive path planning algorithm is used to control the robot to accurately place the screw to the assembly position, while considering obstacle avoidance and energy consumption optimization; during the screw assembly process, an adaptive control strategy is used to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screw; the screw posture is evaluated in real time through a machine vision system and a posture estimation algorithm, and if an abnormal screw posture is detected, an automatic correction mechanism is triggered; The third unit is used to process and analyze the data generated during the detection and loading process in real time through edge computing equipment, and continuously optimize the multimodal deep learning model using an incremental learning algorithm to form a closed-loop control.

[0077] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0078] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0079] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A screw defect detection and feeding control method based on machine vision, characterized in that: include: Acquire multi-angle images of the screws through a machine vision system, input the multi-angle images into a pre-trained multi-modal deep learning model, identify the type and position of defects on the screw surface, and extract geometric dimension information of the screws, wherein the machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multi-modal deep learning model integrates a convolutional neural network and a recurrent neural network; evaluate the quality of the screws according to the identified defect type, position, and geometric dimension information, combined with a preset quality standard, and obtain the quality grade of the screws; Based on the quality level, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; Use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw transportation process, use a high-speed camera system to dynamically track the screw posture, combine the Kalman filter algorithm to predict the motion trajectory of the screw, and record the real-time position and posture information of each screw; Based on the real-time position and posture information of the screws, an adaptive path planning algorithm is used to control the robot to accurately place the screws at the assembly position, while taking into account obstacle avoidance and energy consumption optimization. During the screw assembly process, an adaptive control strategy is used to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screws. The screw posture is evaluated in real time through a machine vision system and posture estimation algorithm. If an abnormal screw posture is detected, an automatic correction mechanism is triggered. The data generated during the detection and loading process are processed and analyzed in real time through edge computing devices, and the multimodal deep learning model is continuously optimized using incremental learning algorithms to form a closed-loop control.

2. The method according to claim 1, characterized in that: The multimodal deep learning model integrates the steps of convolutional neural network and recurrent neural network: Constructing a multimodal deep learning model including a convolutional neural network branch and a recurrent neural network branch, wherein the convolutional neural network branch adopts a ResNet-50 network structure, and the recurrent neural network branch adopts a bidirectional long short-term memory network; Using the convolutional neural network branch to process the screw image at each angle to extract spatial features; arranging the spatial features in the order of shooting to form a time series; Using the recurrent neural network branch to process the constructed time series, capturing the dependencies between different angles, and extracting time series features; Designing an adaptive feature fusion module to fuse spatial features and temporal features, including using one-dimensional convolution to adjust the number of channels of the output features of the convolutional neural network branch and the recurrent neural network branch, splicing the adjusted features in the channel dimension, and applying a Squeeze-and-Excitation module to learn the dependencies between channels; Construct a spatiotemporal attention mechanism, where the spatial attention uses one-dimensional convolution and softmax operations to calculate the attention weight of each position on the two-dimensional feature map, and the temporal attention uses a fully connected layer and softmax operations to calculate the attention weights of different time steps; The attention weights obtained by the spatiotemporal attention mechanism are applied to the fused spatial features and temporal features to obtain weighted features.

3. The method according to claim 2, characterized in that The method also includes optimizing and training a multimodal deep learning model: A feature pyramid network is constructed based on weighted features to generate a multi-scale feature map; a multi-task learning head is designed based on the multi-scale feature map, including a branch for defect detection and a branch for size measurement, wherein the defect detection branch includes a region proposal network, a RoI Align module, and a classification and regression head, and the size measurement branch includes a global average pooling layer and a fully connected layer; Design a multi-task joint loss function, including cross entropy loss for defect classification, smooth L1 loss for bounding box regression, mean square error loss and L2 regularization term for size regression, and introduce a learnable uncertainty parameter to dynamically adjust the weights of multi-task losses; Training the multimodal deep learning model using a phased training strategy, including freezing the convolutional neural network backbone to train only the recurrent neural network and multi-task learning head, unfreezing the high layers of the convolutional neural network for fine-tuning, and unfreezing all layers for end-to-end fine-tuning; A cosine annealing learning rate strategy is applied during training, and regularization methods are used to prevent overfitting.

4. The method according to claim 1, characterized in that: Based on the real-time position and posture information of the screws, an adaptive path planning algorithm is used to control the robot to accurately place the screws at the assembly position. The steps of considering obstacle avoidance and energy consumption optimization include: The adaptive path planning algorithm adopts a hybrid algorithm based on the artificial potential field method and the improved fast expanding random tree algorithm; Implement multi-level obstacle avoidance strategies, including: using occupancy grid maps and expansion algorithms to handle static obstacles; using Kalman filters to predict the future motion trajectory of dynamic obstacles; introducing the concept of speed obstacles to handle multi-robot collaborative operation scenarios; Energy consumption optimization, including: using cubic spline interpolation to smooth the generated path; solving the time-optimal trajectory planning problem that satisfies the robot's dynamic constraints and converting it into a quadratic planning problem; establishing a robot joint motor energy consumption model that takes into account copper loss, iron loss, and mechanical loss; Construct a multi-objective optimization function, including three sub-goals: time, energy consumption, and accuracy. The time sub-goal is based on the result of time-optimal trajectory planning, the energy consumption sub-goal is calculated based on the energy consumption model of the robot joint motor, and the accuracy sub-goal considers the deviation between the real-time position and posture information of the screw and the target placement position. Solve the multi-objective optimization function, and generate the robot's screw placement instructions based on the results of the multi-objective optimization, combined with the adaptive path planning algorithm and multi-level obstacle avoidance strategy; During the placement process, the real-time position and posture information of the screws is continuously acquired, and the path planning and obstacle avoidance strategies are dynamically adjusted to achieve adaptive control.

5. The method according to claim 4, characterized in that The steps of the adaptive path planning algorithm using a hybrid algorithm based on an artificial potential field method and an improved rapidly expanding random tree algorithm include: The global path is generated using an improved fast-expanding random tree algorithm, which includes adaptive sampling, heuristic bias and dynamic rewriting. Adaptive sampling dynamically adjusts the sampling density according to the complexity of the environment, heuristic bias introduces a goal-oriented sampling strategy to accelerate expansion to the goal, and dynamic rewriting periodically re-evaluates and optimizes the generated path. Based on the global path, an artificial potential field method with an adaptive potential field function is used to optimize the local path. The adaptive potential field function includes an attractive potential field term and a repulsive potential field term. The attractive potential field guides the robot to move toward the target position, and the repulsive potential field keeps the robot away from obstacles. A random disturbance term is introduced into the adaptive potential field function, and the amplitude of the random disturbance term decays exponentially with time. A local planning strategy with a rolling time domain is used to cope with dynamic environments. Local environment map updates, local path optimization, and global path deviation checks are performed in each control cycle. When it is detected that the deviation between the local path and the global path exceeds a preset threshold, an improved fast-expanding random tree algorithm is triggered to re-plan the global path. The optimized local path is smoothly spliced ​​with the global path to generate a complete motion trajectory.

6. The method according to claim 1, characterized in that In the screw assembly process, the steps of dynamically adjusting the robot's force control parameters and motion trajectory using an adaptive control strategy according to the material, size, and assembly surface characteristics of the screw include: Establish a comprehensive force control model, including a thread contact force model that considers thread geometry and material properties, an axial preload force model based on screw material and size properties, and an assembly surface property model using Hertz contact theory; Based on the comprehensive force control model, the friction coefficient and the preload coefficient are estimated in real time using the recursive least square method; Based on the friction coefficient and preload coefficient, a fuzzy adaptive controller is designed, and the Mamdani reasoning method and the center of gravity method are used to defuzzify and dynamically adjust the force control parameters; a reinforcement learning algorithm based on policy gradient is introduced to optimize the force control parameters; The robot's motion trajectory is parameterized using a B-spline curve, and the motion trajectory is optimized using an improved differential evolution algorithm, wherein the improvements include an adaptive mutation strategy, local search enhancement, and parallel computing acceleration; Based on real-time force feedback, the optimized motion trajectory is fine-tuned online using an iterative learning control method. Apply the optimized force control parameters and motion trajectory to the robot control system to perform the screw assembly task; During the screw assembly process, force feedback information is continuously obtained, the comprehensive force control model parameters are updated, and real-time optimization of the force control parameters and motion trajectory is triggered.

7. The method according to claim 1, characterized in that The machine vision system and posture estimation algorithm are used to evaluate the screw posture in real time. If an abnormal screw posture is detected, the steps to trigger the automatic correction mechanism include: A machine vision system is used to acquire screw images, and the acquired screw images are preprocessed in real time, including image segmentation based on local adaptive threshold, non-local mean denoising, illumination non-uniformity correction based on Retinex algorithm, and sub-pixel edge detection; Based on the preprocessed screw image, a deep learning multi-task network is used to estimate the screw posture, and the deep learning multi-task network includes a detection branch, a segmentation branch and a posture branch; based on the screw posture estimation, an iterative closest point algorithm is used to refine the screw posture and minimize the Euclidean distance between the screw CAD model point cloud and the actual screw point cloud; Define the judgment criteria for screw posture abnormality, including position deviation, angle deviation and thread orientation, and automatically adjust the judgment threshold through Bayesian optimization; The screw posture refinement results are compared with the judgment criteria for screw posture abnormality. When the screw posture abnormality is detected, the automatic correction mechanism is triggered, including path planning and compliance control. During the correction process, visual feedback information is continuously obtained, and the posture estimation and correction strategy are updated in real time.

8. A screw defect detection and feeding control system based on machine vision, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain multi-angle images of the screws through a machine vision system, input the multi-angle images into a pre-trained multi-modal deep learning model, identify the type and position of defects on the surface of the screws, and extract geometric size information of the screws. The machine vision system includes a high-resolution industrial camera, a light source system, and a 3D sensor; the multi-modal deep learning model integrates a convolutional neural network and a recurrent neural network; based on the identified defect type, position, and geometric size information, combined with a preset quality standard, the quality of the screws is evaluated to obtain the quality grade of the screws; Based on the quality level, the sorting mechanism is controlled to transport qualified screws and unqualified screws to corresponding storage areas respectively; The second unit is used to use a robotic arm to grab qualified screws and place them on the feeding conveyor belt. During the screw transportation process, a high-speed camera system is used to dynamically track the screw posture, and the Kalman filter algorithm is combined to predict the motion trajectory of the screw, and the real-time position and posture information of each screw is recorded; Based on the real-time position and posture information of the screws, an adaptive path planning algorithm is used to control the robot to accurately place the screws at the assembly position, while taking into account obstacle avoidance and energy consumption optimization. During the screw assembly process, an adaptive control strategy is used to dynamically adjust the robot's force control parameters and motion trajectory according to the material, size and assembly surface characteristics of the screws. The screw posture is evaluated in real time through a machine vision system and posture estimation algorithm. If an abnormal screw posture is detected, an automatic correction mechanism is triggered. The third unit is used to process and analyze the data generated during the detection and loading process in real time through edge computing equipment, and continuously optimize the multimodal deep learning model using an incremental learning algorithm to form a closed-loop control.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Mechanical arm target capturing method and device based on trajectory prediction control and medium

    CN117733870A

  • Visual mechanical arm material grabbing control method and system based on deep learning

    CN118003340A

  • Intelligent package detection method and system

    CN118195994A

  • Mechanical arm control method and system based on multi-mode driving and storage medium

    CN118752495A

  • Object path planning in a sorting facility

    US20230192418A1

Cited By

  • Working parameter automatic deployment method and device of industrial robot and industrial robot

    CN119974015A

  • Self-adaptive gravity center adjusting method and system for transfer robot

    CN119974028A

  • Automatic sorting method and system based on artificial intelligence

    CN120133163A

  • Method for extracting original information of earth-rock mixed material structure

    CN120259132A

  • Deep learning-based tiny target defect identification model training method

    CN120451160A